Image processing device and method

ABSTRACT

The present invention relates to an image processing device and method whereby deterioration of effects of filter processing due to local control of filter processing when encoding or decoding can be suppressed. 
     A boundary control flag generating unit  132  of a control information generating unit  112  generates boundary control flags based on system specification information which a system specification managing unit  141  manages. A control unit  171  of a adaptive filter processing unit  113  determines a processing method for filter processing to be performed as to pixels nearby a slice boundary following the value of the boundary control flag. For example, selection is made to perform filter processing straddling slices or to perform filter processing closed at the present slice.
         The present invention can be applied to an image processing device, for example.

TECHNICAL FIELD

The present invention relates to an image processing device and method,and specifically relates to an image processing device and method whichenable suppression in deterioration of the effects of filter processingdue to local control of filter processing when encoding or whendecoding.

BACKGROUND ART

In recent years, there have come into widespread use devices, compliantto formats such as MPEG (Moving Picture Experts Group) or the like,which handle image information as digital signals, and take advantage ofredundancy peculiar to the image information in order to perform highlyeffective information transmission and storage at that time, to compressthe image by orthogonal transform such as discrete cosine transform orthe like and motion compensation, as both information distribution suchas broadcasting and information reception in general households.

In particular, MPEG2 (ISO (International Organization forStandardization)/IEC (International Electrotechnical Commission)13818-2) is defined as a general-purpose image encoding format, and is astandard encompassing both of interlaced scanning images andsequential-scanning images, and standard resolution images and highdefinition images. For example, MPEG2 has widely been employed now bybroad range of applications for professional usage and for consumerusage. By employing the MPEG2 compression format, a code amount (bitrate) of 4 through 8 Mbps is allocated in the event of an interlacedscanning image of standard resolution having 720×480 pixels, forexample. Also, by employing the MPEG2 compression format, a code amount(bit rate) of 18 through 22 Mbps is allocated in the event of aninterlaced scanning image of high resolution having 1920×1088 pixels,for example, whereby a high compression rate and excellent image qualitycan be realized.

With MPEG2, high image quality encoding adapted to broadcasting usage isprincipally taken as an object, but a lower code amount (bit rate) thanthe code amount of MPEG1, i.e., an encoding format having a highercompression rate is not handled. According to spread of personal digitalassistants, it has been expected that needs for such an encoding formatwill be increased from now on, and in response to this, standardizationof the MPEG4 encoding format has been performed. With regard to an imageencoding format, the specification thereof was confirmed asinternational standard as ISO/IEC 14496-2 in December in 1998.

Further, in recent years, standardization of a standard called H.26L(ITU-T (ITU Telecommunication Standardization Sector) Q6/16 VCEG (VideoCoding Experts Group)) has progressed, originally intended for imageencoding for videoconferencing usage. With H.26L, it has been known thatas compared to a conventional encoding format such as MPEG2 or MPEG4,though greater computation amount is requested for encoding and decodingthereof, higher encoding efficiency is realized. Also, currently, aspart of activity of MPEG4, standardization for also taking advantage offunctions not supported by H.26L with this H.26L taken as a base, torealize higher encoding efficiency, has been performed as Joint Model ofEnhanced-Compression Video Coding. As a schedule of standardization,H.264 and MPEG-4 Part10 (AVC (Advanced Video Coding)) become aninternational standard in March, 2003.

Also, there is adaptive loop filter (ALF (Adaptive Loop Filter)) as anext generation video encoding technique which is being considered as ofrecent (see NPL 1 for example). According to this adaptive loop filter,optimal filter processing is performed each frame, and block noise whichwas not completely removed at the deblocking filter, and noise due toquantization, can be reduced.

However, images generally have various features, so optimal filtercoefficients are locally different. With the method in NPL 1, the samefilter coefficient is applied to all pixels within one frame, so theimage quality of the overall frame improves, but there has been theconcern that there may be local deterioration.

Accordingly, there has been conceived not performing filter processingin regions which locally deteriorate (see NPL 2 and NPL 3, for example).In this case, the image encoding device corresponds multiple controlblocks arrayed without gaps as if they were being used for paving, withregions of the image, and controls whether or not to perform filterprocessing on the image for each control block. The image encodingdevice sets flag information for each block, and performs adaptivefilter processing according to the flag information. In the same way,the image decoding device also performs adaptive filter processingaccording to the flag information.

CITATION LIST Non Patent Literature

-   NPL 1: Yi-Jen Chiu and L. Xu, “Adaptive (Wiener) Filter for Video    Compression,” ITU-T SG16 Contribution, C437, Geneva, April 2008.-   NPL 2: Takeshi. Chujoh, et al., “Block-based Adaptive Loop Filter”    ITU-T SG16 Q6 VCEG Contribution, AI18, Germany, July, 2008-   NPL 3: T. Chujoh, N. Wada and G. Yasuda, “Quadtree-based Adaptive    Loop Filter,” ITU-T SG16 Q6 VCEG Contribution, VCEG-AK22(r1), Japan,    April, 2009

SUMMARY OF INVENTION Technical Problem

However, there is a method in which one frame is divided into multipleslices, and encoding processing and decoding processing of the image isperformed for each such slice (multi-slice). NPL 2 and NPL 3 make nomention regarding processing of pixels near boundaries of slices in sucha multi-slice case, and how this should be processed has been unclear.

The present invention has been proposed in light of this situation, andit is an object thereof to suppress deterioration of the effects offilter processing due to local control of filter processing whenencoding or when decoding.

Solution to Problem

One aspect of the present invention is an image processing deviceincluding: determining means configured to determine whether or notthere are included, in surrounding pixels of a pixel to be processed byfilter processing locally performed on an image, pixels of a sliceneighboring a slice in which the pixel to be processed is included;selecting means configured to select, from a plurality of methods, amethod for the filter processing to be performed on the pixel to beprocessed, based on a boundary control flag, in the event thatdetermination has been made by the determining means that a pixel of theneighboring slice is included in the surrounding pixels; and filterprocessing means configured to perform the filter processing as to thepixel to be processed with the method selected by the selecting means.

The selecting means may select one of a method to perform the filterprocessing on the pixel to be processed after the surrounding pixelssituated in the neighboring slice have been obtained, and a method toperform the filter processing on the pixel to be processed by generatingdummy data of the surrounding pixels situated in the neighboring sliceby duplicating the surrounding pixels situated in the slice includingthe pixel to be processed.

The selecting means may select one of a method to perform the filterprocessing on the pixel to be processed after the surrounding pixelssituated in the neighboring slice have been obtained, and a method toomit performing the filter processing on the pixel to be processed.

The image processing device may further include: generating meansconfigured to generate the boundary control flag based on systemspecifications; with the selecting means selecting a method of thefilter processing as to the pixel to be processed, based on the boundarycontrol flag generated by the generating means.

The system specifications may include hardware resources of the imageprocessing device.

The system specifications may include the usage purpose of the imageprocessing device.

The image processing device may further include: encoding meansconfigured to encode the image and generate encoded data; with theencoding means further encoding the boundary control flag generated bythe generating means, and adding to the encoded data.

The image processing device may further include: decoding meansconfigured to decode encoded data of the image having been encoded, andgenerate the image; with the decoding means further decoding the encodedboundary control flag which has been added to the encoded data; and theselecting means selecting a method for the filter processing as to thepixel to be processed, based on the boundary control flag decoded by thedecoding means.

One aspect of the present invention also is an image processing methodwherein determining means of an image processing device determinewhether or not there are included, in surrounding pixels of a pixel tobe processed by filter processing locally performed on an image, pixelsof a slice neighboring a slice in which the pixel to be processed isincluded, selecting means of the image processing device select, from aplurality of methods, a method for the filter processing to be performedon the pixel to be processed, based on a boundary control flag, in theevent that determination has been made that a pixel of the neighboringslice is included in the surrounding pixels, and filter processing meansof the image processing device perform the filter processing as to thepixel to be processed with the method that has been selected.

With an aspect of the present invention, determination is made regardingwhether or not there are included, in surrounding pixels of a pixel tobe processed by filter processing locally performed on an image, pixelsof a slice neighboring a slice in which the pixel to be processed isincluded, a method for the filter processing to be performed on thepixel to be processed is selected from a plurality of methods, based ona boundary control flag, in the event that determination has been madethat a pixel of the neighboring slice is included in the surroundingpixels, and the filter processing is performed as to the pixel to beprocessed with the method that has been selected.

Advantageous Effects of Invention

According to the present invention, an image can be encoded or decoded.Particularly, deterioration of effects of filter processing due to localcontrol of filter processing when encoding or decoding can besuppressed. For example, the deterioration in the effects of filterprocessing can be suppressed even in case of performing encoding ordecoding with each frame of an image divided into a plurality.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of anembodiment of an image encoding device to which the present inventionhas been applied.

FIG. 2 is a diagram describing variable block size motionprediction/compensation processing.

FIG. 3 is a block diagram illustrating a primary configuration exampleof a control information generating unit.

FIG. 4 is a diagram describing ALF blocks and filter block flags.

FIG. 5 is a diagram describing an example of multi-slice.

FIG. 6 is a diagram describing surrounding pixels used for filterprocessing.

FIG. 7 is a diagram describing the way in which filter processing isperformed close to a boundary.

FIG. 8 is a block diagram illustrating a primary configuration exampleof an adaptive filter processing unit.

FIG. 9 is a flowchart describing an example of the flow of encodingprocessing.

FIG. 10 is a flowchart describing an example of the flow of controlinformation generating processing.

FIG. 11 is a flowchart describing an example of the flow of boundarycontrol flag setting processing.

FIG. 12 is a flowchart describing an example of the flow of adaptivefilter control processing.

FIG. 13 is a flowchart describing an example of the flow of filterprocessing.

FIG. 14 is a flowchart describing an example of the flow of filterprocessing.

FIG. 15 is a block diagram illustrating a primary configuration exampleof an image decoding device to which the present invention has beenapplied.

FIG. 16 is a flowchart describing an example of the flow of decodingprocessing.

FIG. 17 is a block diagram illustrating another configuration example ofan image encoding device to which the present invention has beenapplied.

FIG. 18 is a block diagram illustrating another configuration example ofan image decoding device to which the present invention has beenapplied.

FIG. 19 is a flowchart describing an example of the flow of processingfor exchanging specification information.

FIG. 20 is a diagram describing another example of ALF blocks and filterblock flags.

FIG. 21 is a diagram describing another example of ALF blocks and filterblock flags.

FIG. 22 is a diagram describing the way of processing is performed inthe case of multi-slice.

FIG. 23 is a block diagram illustrating a primary configuration exampleof a personal computer to which the present invention has been applied.

FIG. 24 is a block diagram illustrating a principal configurationexample of a television receiver to which the present invention has beenapplied.

FIG. 25 is a block diagram illustrating a principal configurationexample of a cellular telephone to which the present invention has beenapplied.

FIG. 26 is a block diagram illustrating a principal configurationexample of a hard disk recorder to which the present invention has beenapplied.

FIG. 27 is a block diagram illustrating a principal configurationexample of a camera to which the present invention has been applied.

FIG. 28 is a diagram illustrating an example of macro blocks.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described.Note that description will proceed in the following order.

1. First Embodiment (image encoding device)2. Second Embodiment (image decoding device)3. Third Embodiment (image encoding/decoding system)

4. Fourth Embodiment (QALF)

5. Fifth Embodiment (personal computer)6. Sixth Embodiment (television receiver)7. Seventh Embodiment (cellular telephone)8. Eighth Embodiment (hard disk recorder)9. Ninth Embodiment (camera)

1. First Embodiment [Configuration of Device]

FIG. 1 represents the configuration of an embodiment of an imageencoding device serving as an image processing device to which thepresent invention has been applied.

An image encoding device 100 shown in FIG. 1 is an image encoding devicewhich subjects an image to compression encoding using, for example, theH.264 and MPEG-4 Part10 (Advanced Video Coding) (hereafter, written asH.264/AVC) format, and further employs an adaptive loop filter.

With the example in FIG. 1, the image encoding device 100 has an A/D(Analog/Digital) conversion unit 101, a screen rearranging buffer 102, acomputing unit 103, an orthogonal transform unit 104, a quantizationunit 105, a lossless encoding unit 106, and a storing buffer 107. Theimage encoding device 100 also has an inverse quantization unit 108, aninverse orthogonal transform unit 109, a computing unit 110, and adeblocking filter 111. Further, the image encoding device 100 has acontrol information generating unit 112, an adaptive filter processingunit 113, and frame memory 114. Also, the image encoding device 100 hasan intra prediction unit 115, a motion compensation unit 116, a motionprediction unit 117, and a prediction image selecting unit 118. Further,the image encoding device 100 has a rate control unit 119.

The A/D conversion unit 101 performs A/D conversion of an input image,and outputs to the screen rearranging buffer 102 and stores. The screenrearranging buffer 102 rearranges the images of frames in the storedorder for display into the order of frames for encoding according to GOP(Group of Picture). The computing unit 103 subtracts from the image readout from the screen rearranging buffer 102 the prediction image from theintra prediction unit 115 selected by the prediction image selectingunit 118 or the prediction image from the motion compensation unit 116,and outputs difference information thereof to the orthogonal transformunit 104. The orthogonal transform unit 104 subjects the differenceinformation from the computing unit 103 to orthogonal transform, such asdiscrete cosine transform, Karhunen-Loéve transform, or the like, andoutputs a transform coefficient thereof. The quantization unit 105quantizes the transform coefficient that the orthogonal transform unit104 outputs.

The quantized transform coefficient that is the output of thequantization unit 105 is input to the lossless encoding unit 106, whereit is subjected to lossless encoding, such as variable length coding,arithmetic coding, or the like, and compressed.

The lossless encoding unit 106 obtains information indicating intraprediction and so forth from the intra prediction unit 115, and obtainsinformation indicating an inter prediction mode, and so forth from themotion prediction unit 117. Note that the information indicating intraprediction will also be referred to as intra prediction mode informationhereinafter. Also, the information indicating inter prediction will alsobe referred to as inter prediction mode information hereinafter.

The lossless encoding unit 106 obtains control information of adaptivefilter processing performed at the adaptive filter processing unit 113from the control information generating unit 112.

The lossless encoding unit 106 encodes the quantized transformcoefficient, and also encodes the control information of adaptive filterprocessing, the information indicating intra prediction, the informationindicating an inter prediction mode, quantization parameters, and soforth, and takes these as part of header information in the compressedimage (multiplexes). The lossless encoding unit 106 supplies the encodeddata to the storing buffer 107 for storage.

For example, with the lossless encoding unit 106, lossless encodingprocessing, such as variable length coding, arithmetic coding, or thelike, is performed. Examples of the variable length coding include CAVLC(Context-Adaptive Variable Length Coding) determined by the H.264/AVCformat. Examples of the arithmetic coding include CABAC(Context-Adaptive Binary Arithmetic Coding).

The storing buffer 107 temporarily holds the data supplied from thelossless encoding unit 106, and at a predetermined timing outputs thisto, for example, a storage device or transmission path or the likedownstream not shown in the drawing, as a compressed image encoded bythe H.264/AVC format.

Also, the quantized transform coefficient output from the quantizationunit 105 is also input to the inverse quantization unit 108. The inversequantization unit 108 performs inverse quantization of the quantizedtransform coefficient with a method corresponding to quantization at thequantization unit 105, and supplies the obtained transform coefficientto the inverse orthogonal transform unit 109.

The inverse orthogonal transform unit 109 performs inverse orthogonaltransform of the supplied transform coefficients with a methodcorresponding to the orthogonal transform processing by the orthogonaltransform unit 104. The output subjected to inverse orthogonal transformis supplied to the computing unit 110. The computing unit 110 adds theprediction image supplied from the prediction image selecting unit 118to the inverse orthogonal transform result supplied from the inverseorthogonal transform unit 109, i.e., the restored differenceinformation, and obtains a locally decoded image (decoded image). Theaddition results thereof are supplied to the deblocking filter 111.

The deblocking filter 111 removes block noise from the decoded image.The deblocking filter 111 then supplies the noise removal results to thecontrol information generating unit 112 and the adaptive filterprocessing unit 113.

The control information generating unit 112 obtains the decoded imagesupplied from the deblocking filter 111 and the current input image readout from the screen rearranging buffer 102, and generates from thesecontrol information for adaptive filtering to be performed at theadaptive filter processing unit 113. While details will be describedlater, the control information includes filter coefficients, block size,filter block flags, and boundary control flags and the like.

The control information generating unit 112 supplies the generatedcontrol information to the adaptive filter processing unit 113. Thecontrol information generating unit 112 also supplies the generatedcontrol information to the lossless encoding unit 106 as well. Asdescribed above, the control information is subjected to losslesscompression processing by the lossless encoding unit 106, and includedin the image compressed information (multiplexed). That is to say, thecontrol information is sent to the image decoding device along with theimage compression information.

The adaptive filter processing unit 113 performs filter processing onthe decoded image supplied form the deblocking filter 111, using thefilter coefficients, block size specification, and filter block flagsand the like, of the control information supplied from the controlinformation generating unit 112. A Wiener filter (Wiener Filter), forexample, is used as this filter. Of course, a filer other than a Wienerfilter may be used. The adaptive filter processing unit 113 supplies thefilter processing results to the frame memory 114, and stores as areference image.

The frame memory 114 outputs the stored reference image to the motioncompensation unit 116 and motion prediction unit 117 at a predeterminedtiming.

With this image encoding device 100, the I picture, B picture, and Ppicture from the screen rearranging buffer 102 are supplied to the intraprediction unit 115 as an image to be subjected to intra prediction(also referred to as intra processing), for example. Also, the B pictureand P picture read out from the screen rearranging buffer 102 aresupplied to the motion compensation unit 117 as an image to be subjectedto inter prediction (also referred to as inter processing).

The intra prediction unit 115 performs intra prediction processing ofall of the candidate intra prediction modes based on the image to besubjected to intra prediction read out from the screen rearrangingbuffer 102, and the reference image supplied from the frame memory 114to generate a prediction image.

With the intra prediction unit 115, information relating to the intraprediction mode applied to the current block/macroblock is transmittedto the lossless encoding unit 106, and is encoded as a part of theheader information in the image compression information. With the H.264image information encoding format, the intra 4×4 prediction mode, intra8×8 prediction mode, and intra 16×16 prediction mode are defined forluminance signals, and also with regard to color difference signals, aprediction mode can be defined for each macroblock, independent from theluminance signals. For the intra 4×4 prediction mode, one intraprediction mode is defined for each 4×4 luminance block. For the intra8×8 prediction mode, one intra prediction mode is defined for each 8×8luminance block. For the intra 16×16 prediction mode and colordifference signals, one intra prediction mode is defined for eachmacroblock.

The intra prediction unit 115 calculates a cost function value as to theintra prediction mode where the prediction image has been generated, andselects the intra prediction mode where the calculated cost functionvalue gives the minimum value, as the optimal intra prediction mode. Theintra prediction unit 115 supplies the prediction image generated in theoptimal intra prediction mode to the prediction image selecting unit118.

With regard to the image to be subjected to inter encoding, the motionprediction unit 117 obtains image information supplied from the screenrearranging buffer 102 (input image) and image information serving asthe reference frame supplied from the frame memory 114 (decoded image),and calculates a motion vector. The motion prediction unit 117 suppliesmotion vector information indicating the calculated motion vector to thelossless encoding unit 106. This motion vector information is subjectedto lossless compression processing by the lossless encoding unit 106,and included in the image compressing information. That is to say themotion vector information is sent to the image decoding device alongwith the image compression information.

Also, the motion prediction unit 117 also supplies the motion vectorinformation to the motion compensation unit 116.

The motion compensation unit 116 performs motion compensation processingin accordance with the motion vector information supplied from themotion prediction unit 117, and generates inter prediction imageinformation. The motion compensation unit 116 supplies the generatedprediction image information to the prediction image selecting unit 118.

In the case of an image for performing intra encoding, the predictionimage selecting unit 118 supplies the output of the intra predictionunit 115 to the computing unit 103, and in the event of an image forperforming inter encoding, supplies the output of the motioncompensation unit 116 to the computing unit 103.

The rate control unit 119 controls the rate of quantization operationsof the quantization unit 105 based on the compressed image stored in thestoring buffer 107, such that overflow or underflow does not occur.

With MPEG (Moving Picture Experts Group) 2, the increments of motionprediction/compensation processing is motion compensation blocks, andindependent motion vector information can be held at each motioncompensation block. The size of a motion compensation block is 16×16pixels in the case of frame motion compensation mode, and in the case offield motion compensation mode is 16×8 pixels for each of the firstfield and the second field.

On the other hand, with AVC (Advanced Video Coding), one macroblockconfigured of 16×16 pixels, as shown at the upper side in FIG. 2, can bedivided into any of the partitions of 16×16, 16×8, 8×16, or 8×8, witheach holding independent motion vector information. Also, as shown atthe lower side in FIG. 2, a 8×8 partition can be divided into any of thesub partitions of 8×8, 8×4, 4×8, or 4×4, with each holding independentmotion vector information. Motion prediction/compensation processing isperformed with this motion compensation block as an increment.

FIG. 3 is a block diagram illustrating a primary configuration exampleof the control information generating unit 112.

The control information generating unit 112 generates controlinformation used at the adaptive filter (ALF (Adaptive Loop Filter))which is a loop filter, performed at the adaptive filter processing unit113. The control information generating unit 112 generates, as thecontrol information, filter coefficients, ALF block size, filter blockflags, and boundary control flags, for example.

The control information generating unit 112 has a filter coefficientcalculating unit 131, boundary control flag generating unit 132, andblock information generating unit 133.

The filter coefficient calculating unit 131 obtains the decoded imagesupplied from the deblocking filter 111 and current input image read outfrom the screen rearranging buffer 102, and calculates an ALF filtercoefficient for each frame.

The boundary control flag generating unit 132 generates a boundarycontrol flag (alf_enable_in_slice_boundary) which controls how filterprocessing is to be performed as to pixels near the boundary of slices,of which a plurality is formed in the frame (specifies filter processingmethod). Details will be described later.

The block information generating unit 133 determines the ALF block sizebased on the decoded image supplied from the deblocking filter 111 andthe filter coefficients calculated by the filter coefficient calculatingunit 131, and generates a filter block flag for each ALF block withinthe slice to be processed.

Now, description will be made regarding the ALF block and filter blockflag. FIG. 4 is a diagram for describing ALF blocks and filter blockflags.

As described above, the adaptive filter has filter coefficients set foreach frame. That is to say, optimal filter processing is performed inincrements of frames. However, generally, frame images are not uniformoverall, and have various features locally. Therefore, optimal filtercoefficients differ locally. Accordingly, while the filter processingusing filter coefficients determined each frame as described aboveimprove the image quality for the overall frame, there has been concernthat this will in fact deteriorate locally.

Accordingly, BALF (Block based Adaptive Loop Filter) in which filterprocessing is not performed at regions where image quality locallydeteriorates, has been conceived.

A decoded image following deblocking filter processing is shown in frame151 in A in FIG. 4. As shown in B in FIG. 4, the block informationgenerating unit 133 arrays multiple ALF blocks 152, which are controlblocks serving as the increment of control for adaptive filterprocessing locally performed, without gaps as if they were being usedfor paving the entire region of the frame 151. The region where the ALFblocks 152 are placed does not have to be the same as the region of theframe 151, but includes at least the entire region of the frame. Theregion of the frame 151 is resultantly divided into the regions of theALF blocks 152 (multiple regions).

The block information generating unit 133 determines the horizontaldirection size (both-sided arrow 153) and vertical direction size(both-sided arrow 154) of the ALF blocks 152. For the size of the ALFblocks, one of 8×8, 16×16, 24×24, 32×32, 48×48, 64×64, 96×96, or128×128, can be specified for each slice. The information specifying thesize of the ALF block will be called block size index.

Once the block size is decided, the number of ALF blocks per frame hasalso been decided, since the frame size is fixed.

As shown in C in FIG. 4, the block information generating unit 133 setsa filter block flag 155 which controls whether or not to perform filterprocessing, in each Alf block 152. For example, a filter block flag 155with a value of “1” is generated for a region where the image quality isimproved by the adaptive filter, and a filter block flag 155 with avalue of “0” is generated for a region where the image quality isdeteriorated by the adaptive filter. With the filter block flag 155, thevalue of “1” is a value indicating that filter processing is to beperformed, and the value of “0” is a value indicating that filterprocessing is not to be performed.

The adaptive filter processing unit 113 controls the adaptive filterprocessing based on the value of the filter block flag 155. For example,the adaptive filter processing unit 113 performs filter processing onlyat the regions where the ALF blocks 152 have a value of “1” for thefilter flag 155, and does not perform filter processing at the regionswhere the ALF blocks 152 have a value of “0” for the filter flag 155.

Also, the above-described block size index and filter block flag areincluded in the slice header of the image compression information, andsent from the image encoding device 100 to the image decoding device.The one or more filter block flags corresponding to the number of ALFblocks are included in the slice header in the order of raster scan, forexample.

Accordingly, the smaller the size of the ALF block, the finer filtercontrol can be realized, and more appropriate ALF filtering can beperformed. However, smaller ALF block size increases the bit amount ofthe filter block flags. That is to say, the smaller the ALF block sizeis the more the encoding efficiency of the image compression informationdecreases. Thus, the capabilities of the adaptive filter and theencoding efficiency of the image compression information are in atradeoff relation.

The number of ALF blocks is calculated as with the following Expression(1).

$\begin{matrix}\left\lbrack {{Mathematic}\mspace{14mu} {Expression}\mspace{14mu} 1} \right\rbrack & \; \\{N_{ALFBLOCK} = {{{floor}\left\lbrack \frac{{16 \times N_{MBw}} + N_{SIZE} - 1}{N_{SIZE}} \right\rbrack} \times {{floor}\left\lbrack \frac{{16 \times N_{MBh}} + N_{SIZE} - 1}{N_{SIZE}} \right\rbrack}}} & (1)\end{matrix}$

In Expression (1), N_(ALFBLOCK) represents the number of ALF blocks.Also, N_(MBw) represents the number of macro blocks in the horizontaldirection of the picture, and N_(MBh) represents the number of macroblocks in the vertical direction of the picture. Further, N_(SIZE)represents the size of one side of an ALF block. Also, floor[x] is afunction where x is rounded off to the decimal so as to be an integer.

Now, with H.264/AVC, once frame can be divided into multiple slices, andimage compression information can be output for each slice. FIG. 5 is adiagram for describing an example of multi-slice. In the case of theexample in FIG. 5, the frame 151 is divided into the three slices ofslice 0, slice 1, and slice 2.

By outputting image compression information in finer slice incrementsthan frames, the image encoding device can generate and output imagecompression information at shorter intervals. That is to say, the imagedecoding device which decodes the image compression information canstart decoding of the image compression information at an earlier stage.That is to say, the delay time from the image being input to encodingprocessing and encoding processing being performed and the image beingoutput can be shortened.

NPL 2 which describes BALF does not disclose this multi-slice. That isto say, only setting ALF blocks for the entire frame is described.However, in the case of multi-slice, there are cases where normal filterprocessing cannot be performed as to pixels near the boundary of slices.

FIG. 6 is a diagram illustrating the way in which filter processing isperformed as to pixels near the slice boundary. In the case ofperforming filter processing as to the pixels to be processed, theadaptive filter processing unit 113 performs this using pixels within apredetermined range around the pixel to be processed (surroundingpixels). For example, in the case of FIG. 6, the adaptive filterprocessing unit 113 performs filter processing on a pixel to beprocessed 161 using 9×9 surrounding pixels 162 shown with hatching.

However, as shown in FIG. 6, the pixel to be processed 161 is a pixelnear the slice boundary 163. Now, the slice boundary 163 indicates theboundary between a slice to be currently processed (current slice) and aslice neighboring the slice to be processed (neighboring slice). That isto say, the slice boundary 163 indicates the outer frame of the currentslice.

As shown in FIG. 6, surrounding pixels 162 used for filter processing ofthe pixel to be processed 161 near the slice boundary 163 partiallycross over the slice boundary 163 and straddle the region of theneighboring slice. That is to say, in order to perform filter processingof the pixel to be processed 161 in the same way as with a normal case,the pixel values of the neighboring slice are necessary, as shown in Ain FIG. 7 for example.

In the case of the example in A in FIG. 7, the adaptive filterprocessing unit 113 performs filter processing as to the pixel EE whichis the pixel to be processed, using the pixel AA through pixel JJ inboth the current slice and the neighboring slice.

However, in order to do this, generating of the decoded image of theneighboring slice needs to be waited for. Accordingly, in this case,there has been the concern that the delay time of encoding processingwould increase.

On the other hand, there is a method for generating and using dummydata, as shown in B in FIG. 7, for example. In the case of the examplein B in FIG. 7, the adaptive filter processing unit 113 duplicates thepixel EA through pixel EJ adjacent to the slice boundary 163, therebygenerating pixels within the neighboring slice for the surroundingpixels 162 (dummy data). The adaptive filter processing unit 113performs filter processing as to the pixel EE using the generated dummydata.

Thus, the adaptive filter processing unit 113 does not need to wait forpixels of the neighboring slice to be generated, and filter processingof the pixel EE can be performed at an earlier stage than with the caseof A in FIG. 7.

That is to say, with the case of the method of A in FIG. 7 using thepixels of the neighboring slice, the adaptive filter processing unit 113uses actual data, so filter processing can be performed more suitablefor the contents of the actual image. That is to say, great improvementof image quality due to filter processing can be expected.

On the other hand, in the case of the method in B in FIG. 7, theadaptive filter processing unit 113 does not need data of the adjacentslice for filter processing, and processing can be performed with thedata of the current slice alone, so filter processing can be performedat an earlier stage.

Which method is desirable differs depending on the systemspecifications, user requests, and so forth. For example, if the systememphasizes image quality, the method shown in A in FIG. 7, but themethod in A in FIG. 7 consumes a greater amount of memory than themethod in B in FIG. 7, and there is the concern that the delay time willincrease. Accordingly, depending on the memory capacity which can beused with the system and the tolerable delay time length, there may becauses where the method of B in FIG. 7 is more desirable.

The boundary control flag controls the method of filter processing as tosuch pixels near a boundary.

Returning to FIG. 3, the boundary control flag generating unit 132generates such boundary control flags. The boundary control flaggenerating unit 132 has a system specification managing unit 141, adetermining unit 142, and a generating unit 143.

The system specification managing unit 141 manages the specifications ofthe system performing image processing (hardware resources, usagepurpose, etc.) including the image encoding device 100. For example, thesystem specification managing unit 141 may be arranged to manage thespecifications (hardware resources, usage purpose, etc.) of the imagedecoding device encoded at the image encoding device 100.

The determining unit 142 determines whether or not the pixel to beprocessed is a pixel near the boundary. The generating unit 143generates boundary control flags for the pixels to be processed whichhave been determined to be pixels near the boundary.

FIG. 8 is a block diagram illustrating a primary configuration exampleof the adaptive filter processing unit 113 in FIG. 1.

The adaptive filter processing unit 113 performs filter processing onthe decoded image supplied from the deblocking filter 111 using thecontrol information supplied from the control information generatingunit 112.

As shown in FIG. 9, the adaptive filter processing unit 113 has acontrol unit 171, an adaptive filter 172, and a selecting unit 173.

The control unit 171 controls the adaptive filter 172 and the selectingunit 173. For example, the control unit 171 obtains control informationfrom the control information generating unit 112, and controls thefilter processing based on this control information.

The adaptive filter 172 performs filter processing of a region in thedecoded image supplied from the deblocking filter 111, specified as ALFblocks to be processed from the control unit 171, using a filtercoefficient set by the control unit 171.

The adaptive filter 172 has a buffer 181, an in-slice adaptive filter182, a first adaptive filter for boundary 183, and a second adaptivefilter for boundary 184.

The buffer 181 temporarily holds a decoded image supplied from thedeblocking filter 111. The buffer 181 can hold not only the slice to beprocessed, but also the slice neighboring the slice to be processed(neighboring slice).

The in-slice adaptive filter 182 performs filter processing as to pixelsto be processed which are not near the slice boundary and regardingwhich pixels of the neighboring slice are not included in thesurrounding pixels, under control of the control unit 171. That is tosay, the in-slice adaptive filter 182 performs filter processing usingonly pixels of the current slice.

The first adaptive filter for boundary 183 performs filter processingstraddling slices on pixels to be processed which are near the sliceboundary and regarding which pixels of the neighboring slice areincluded in the surrounding pixels, under control of the control unit171. That is to say, the first adaptive filter for boundary 183 performsfilter processing using the pixels of the current slice and neighboringslice, with a method such as shown in A in FIG. 7. Accordingly, thefirst adaptive filter for boundary 183 starts filter processing afterpixels of the adjacent slice have been accumulated in the buffer 181.

The second adaptive filter for boundary 184 performs filter processingclosed to the current slice, on pixels to be processed which are nearthe slice boundary and regarding which pixels of the neighboring sliceare included in the surrounding pixels, under control of the controlunit 171. That is to say, the second adaptive filter for boundary 184performs filter processing by generating dummy data as necessary, usingthe pixels of the current slice alone, with a method such as shown in Bin FIG. 7. Accordingly, the second adaptive filter for boundary 184starts filter processing upon pixels of the current slice beingaccumulated in the buffer 181.

The control unit 171 selects one of the in-slice adaptive filter 182,first adaptive filter for boundary 183, and second adaptive filter forboundary 184, following the position of the pixel to be processed andthe value of the boundary control flag included in the controlinformation, and causes the selected processing unit to execute filterprocessing with its own method.

Also, the control unit 171 controls the filter processing start timingof the selected processing unit (in-slice adaptive filter 182, firstadaptive filter for boundary 183, or second adaptive filter for boundary184), in accordance with the accumulation state of the image in thebuffer 181.

The adaptive filter 172 (in-slice adaptive filter 182, first adaptivefilter for boundary 183, or second adaptive filter for boundary 184)supplies the filter processing results to the selecting unit 173.

Under control of the control unit 171, the selecting unit 173 selectsone of the decoded image supplied from the deblocking filter 111(decoded image not subjected to adaptive filter processing) and thedecoded image supplied from the adaptive filter 172 (decoded imagesubjected to adaptive filter processing), supplies this to the framememory 114, and stores as a reference image.

The control unit 171 controls the selecting unit 173 following the valueof the filter block flag included in the control information to selectone of the decoded image not subjected to adaptive filter processing andthe decoded image subjected to adaptive filter processing.

That is to say, the adaptive filter processing unit 113 performs filterprocessing only for a region in the decoded image supplied from thedeblocking filter 111 regarding which indication has been made toperform filter processing by the filter block flag (region regardingwhich determination has been made that image quality will be improved byfilter processing).

[Flow of Processing]

Next, the flow of processing using the portions configured as describedabove will be described. First, an example of the low of encodingprocessing performed by the image encoding device 100 will be describedwith reference to the flowchart in FIG. 9.

In step S101, the A/D conversion unit 101 converts an input image fromanalog to digital. In step S102, the screen rearranging buffer 102stores the A/D converted image, and performs rearranging from thesequence for displaying the pictures to the sequence for encoding.

In step S103, the computing unit 103 computes difference between animage rearranged by the processing in step S102 and the predictionimage. The prediction image is supplied to the computing unit 103 fromthe motion compensation unit 116 in the event of performing interprediction, and from the intra prediction unit 115 in the event ofperforming intra prediction, via the prediction image selecting unit118.

The difference data is smaller in the data amount as compared to theoriginal image data. Accordingly, the data amount can be compressed ascompared to the case of encoding the original image without change.

In step S104, the orthogonal transform unit 104 subjects the differenceinformation generated by the processing in step S103 to orthogonaltransform.

Specifically, orthogonal transform, such as discrete cosine transform,Karhunen-Loéve transform, or the like, is performed, and a transformcoefficient is output. In step S105, the quantization unit 105 quantizesthe transform coefficient. At the time of this quantization, a rate iscontrolled such as later-described processing in step S119 will bedescribed.

The difference information thus quantized is locally decoded as follows.Specifically, in step S106, the inverse quantization unit 108 subjectsthe transform coefficient quantized by the quantization unit 105 toinverse quantization using a property corresponding to the property ofthe quantization unit 105. In step S107, the inverse orthogonaltransform unit 109 subjects the transform coefficient subjected toinverse quantization by the inverse quantization unit 108 to inverseorthogonal transform using a property corresponding to the property ofthe orthogonal transform unit 104.

In step S108 the computing unit 110 adds the prediction image input viathe prediction image selecting unit 118 to the locally decodeddifference information, and generates a locally decoded image (the imagecorresponding to the input to the computing unit 103). In step S109, thedeblocking filter 111 subjects the image output from the computing unit110 to filtering. Thus, block noise is removed.

Upon the above processing being performed for one slice, in step S110the control information generating unit 112 generates controlinformation to be used for adaptive filter processing. The details ofthe control information generating processing will be described later indetail.

Upon control information such as filter coefficients, ALF block size,and filter block flag and the like being generated by the processing instep S110, in step S111 the adaptive filter processing unit 113 performsadaptive filter processing on the decoded image subjected to thedeblocking filter processing in the processing of step S109. Details ofthis adaptive filter processing will be described later.

In step S112, the frame memory 114 stores the image subjected toadaptive filter processing in step S111.

In step S113, the intra prediction unit 115 performs intra predictionprocessing in the intra prediction mode. In step S114, the motionprediction unit 117 and motion compensation unit 116 perform motionprediction/compensation processing in the inter prediction mode.

In step S115, the prediction image selecting unit 118 selects one of aprediction image generated by intra prediction processing or aprediction image generated by inter motion prediction/compensationprocessing, in accordance with the prediction mode of the frame to beprocessed. The prediction image selecting unit 118 supplies the selectedprediction image to the computing units 103 and 110. This predictionimage is, as described above, used for calculations in steps S103 andS108.

In step S116, the lossless encoding unit 106 encodes the quantizedtransform coefficient output from the quantization unit 105.Specifically, the difference image is subjected to lossless encodingsuch as variable length coding, arithmetic coding, or the like, andcompressed. At this time, the lossless encoding unit 106 also encodesthe control information generated in step S110, the intra predictionmode information for intra prediction processing in step S113, the interprediction mode for inter motion prediction/compensation processing instep S114, and so forth.

In step S117, the lossless encoding unit 106 embeds (describes) metadatasuch as the encoded control information and so forth in the sliceheader. This metadata read out and used for when performing imagedecoding. By including (multiplexing) the metadata necessary fordecoding processing in this way, execution of decoding processing isenabled in increments finer than frame increments, and increase of delaytime can be suppressed.

In step S118, the storing buffer 107 stores a difference image as acompressed image. The compressed image stored in the storing buffer 107is read out as appropriate and transmitted to the decoding side via thetransmission path.

In step S119, the rate control unit 119 controls the rate of thequantization operation of the quantization unit 105, so that overflow orunderflow does not occur, based on the compressed image stored in thestoring buffer 107.

Next, description will be made of an example of the flow of controlinformation generating processing executed by the control informationgenerating unit 112 in step S110 in FIG. 10 will be described withreference to the flowchart in FIG. 11.

Upon the control information generating processing being started, thefilter coefficient calculating unit 131 of the control informationgenerating unit 112 calculates a filter coefficient using the inputimage supplied from the screen rearranging buffer 102 and the decodedimage subjected to deblocking filter processing that is supplied fromthe deblocking filter 111. For example, the filter coefficientcalculating unit 131 determines the value of the filter coefficient suchthat the residual of the input image and decoded image is the smallest.

Upon the filter coefficient being calculated, in step S132 the boundarycontrol flag generating unit 132 generates a boundary control flag forcontrolling the adaptive filter processing method as to the pixel nearthe boundary. Details will be described later.

Upon a boundary control flag being generated, in step S133 the blockinformation generating unit 133 generates block information includingALF block size and filter block flag. The ALF block size may bedetermined beforehand, or may be set as appropriate in accordance withthe contents of the image. In this case, the block informationgenerating unit 133 calculates a cost value evaluating the filterprocessing results using a cost function, and determines the ALF blocksize such that the cost value is the smallest, for example.

Also, the block information generating unit 133 determines the value ofthe filter block flag depending on whether the image quality is improvedin the event that the filter processing is applied to the ALF block tobe processed. For example, in the event of determining that imagequality is improved by applying the filter processing, the blockinformation generating unit 133 sets the value of the filter block flagto “1” which indicates that filter processing is to be performed, and inthe event of determining that image quality deteriorates by applying thefilter processing, sets the value of the filter block flag to “0” whichindicates that filter processing is not to be performed.

Upon block information being generated, the flow returns to step S110 inFIG. 9, and processing from step S111 and on is performed.

Note that the calculation of the filter coefficient performed in stepS131 may be performed in frame increments. In this case, the processingin step S131 may be performed only on a predetermined slice within theframe (e.g., a slice where the identification number within the frame isa predetermined value (e.g., “0”), or a slice first processed within theframe, or the like), with this value used for the other slices. Also, anarbitrary image can be used for calculation of filter coefficients. Forexample, calculation may be performed based on past frame images.

Next, an example of the flow of boundary control flag setting processingexecuted in step S132 in FIG. 10 will be described with reference to theflowchart in FIG. 11.

Upon the boundary control flag setting processing being started, in stepS151 the system specification managing unit 141 of the boundary controlflag generating unit 132 obtains system specification information.

This system specification information is information including thehardware resources and usage purpose and so forth of the systemincluding the image encoding device 100, for example. Hardware resourcesare hardware resources of the devices configuring the system (includingthe image encoding device 100), and for example includes processingcapabilities, usable memory capacity, bus transmission speed, and soforth. Also, usage purpose is the operation mode of the overall systemor individual devices, and includes, for example, whether to operatewith emphasis on image quality, whether to operate with emphasis onspeed, and so forth. Of course, information other that these may beincluded in the system specification information.

This system specification information may be stored beforehand in memoryor the like built into the system specification managing unit 141. Inthis case, the system specification managing unit 141 reads out thesystem specification information from the memory by the processing instep S151. Also, at the time of the processing in step S151, the systemspecification managing unit 141 may collect specification informationsuch as described above from parts of the image encoding device 100 andfrom other devices and so forth.

Upon obtaining the system specification information, the systemspecification managing unit 141 supplies this to the determining unit142.

In step S152, the determining unit 142 determines whether or not to usethe next slice for filter processing near the boundary, based on thesupplied system specification information (hardware resources, usagepurpose, etc.). That is to say, in the event that a pixel near aboundary with the neighboring slice being included in surrounding pixelsis the pixel to be processed, the determining unit 142 determineswhether to perform filter processing straddling slices, or to performfilter processing closed at the current slice.

For example, in the event that increased delay time is tolerable, andthere is sufficient memory capacity available at the image encodingdevice 100 and image decoding device and the like, the determining unit142 selects filter processing straddling slices. Also, for example, inthe event that increase in delay time is intolerable or there are notsufficient hardware resources in the devices of the system, thedetermining unit 142 selects filter processing closed at the currentslice.

In the event that determination has been made to use the next slice,i.e., to perform filter processing straddling slices, the flow advancesto step S153. In step S153 the generating unit 143 generates a boundarycontrol flag with a value “1”.

Also, in the event that determination has been made not to use the nextslice, i.e., to perform filter processing closed at the current slice,the flow advances to step S154. In step S154 the generating unit 143generates a boundary control flag with a value “0”.

Upon generating the boundary control flag, the generating unit 143supplies this to the adaptive filter processing unit 113 and losslessencoding unit 106. The lossless encoding unit 106 encodes the boundarycontrol flag supplied from the generating unit 143 as controlinformation, and embeds this in the slice header or the like of thecurrent slice. The adaptive filter processing unit 113 controls adaptivefilter processing using the boundary control flag supplied from thegenerating unit 143.

Upon the processing of step S153 or step S154 ending, the boundarycontrol flag setting processing ends, the flow returns to step S132 inFIG. 10, and processing of step S133 and on is performed.

Next, an example of the flow of adaptive filter processing executed instep S111 in FIG. 9 will be described with reference to the flowchart inFIG. 12.

Upon adaptive filter processing being started, in step S171 the buffer181 obtains the decoded image of the slice to be processed from thedeblocking filter 111. Upon the slice to be processed being obtained, instep S172 the control unit 171 identifies the region of the slice to beprocessed.

In order to know the region of the current slice which is to beprocessed, this can be found by knowing the macroblocks included in thecurrent slice, and knowing the pixels included in the macroblockstherefrom. The control unit 171 obtains the start macroblock address ofthe current slice from the slice header.

Now, the start macroblock address is a number assigned to macroblocks inraster scan order from the upper left of the screen. As shown in FIG. 5,the macroblock address at the upper left in the image (frame 151) is 0.Slice 0 is started from the upper left of the frame 151, so themacroblock address of the start macroblock 156-1 of the slice 0 is 0.Following this order, the end macroblock 156-2 or the slice 0 is E0.Also, in the same way as with this slice 0, the macroblock address ofthe start macroblock 157-1 of slice 1 is S1, and the macroblock addressof the end macroblock 15721 is E1. Further, the macroblock address ofthe start macroblock 158-1 of slice 2 is S2, and the macroblock addressof the end macroblock 158-2 is E2.

As the current slice is decoded, one macroblock address is added eachtime decoding processing of one macroblock is completed, and eventuallythe end macroblock of the current slice is reached. A flag indicatingthe end macroblock of the slice is set at the end macroblock. Due tothese, all macroblock addresses which the current slice holds can beknown. That is to say, this is from the start macroblock address to theend macroblock address.

Now, with a sequence parameter set (SPS (Sequence Parameter Set)) of anAVC stream (image compression information), the image size of one frameis indicated by the number of macroblocks.pic_height_in_map_units_minus1 indicates the number of macroblocks inthe vertical direction of the image. pic_width_in_mbs_minus1 indicatesthe number of macroblocks in the horizontal direction of the image.

Accordingly, from the macroblock address, the position of the macroblockis expressed by the following Expression (2) and Expression (3).

mbx=macro block address%pic_width_in_(—) mbs_minus1  (2)

mby=floor[macro block address/pic_width_in_(—) mbs_minus1]  (3)

In Expression (2) and Expression (3), mbx indicates which number fromthe left the macroblock is, and mby indicates what number from the topthe macroblock is. Also, floor [z] indicates z being rounded out at thedecimal so as to be an integer, and A % B indicates the remainder ofhaving divided A with B.

If we say that the size of the macroblock is determined to be 16×16pixels, the vertical direction and horizontal direction position of thepixel at the upper left of the macroblock is (16×mbx, 16×mby), and thepixels included in the macroblock are pixels included in the range of 16pixels to the lower direction and 16 pixels to the right direction formthe upper left pixel position. Thus far, all pixels of the current slicecan be known. That is to say, the region of the slice to be processed isidentified.

In step S173, the control unit 171 obtains one filter block flaggenerated at the control information generating unit 112.

In step S174, the control unit 171 determines one of unprocessed ALFblocks to be the ALF block to be processed. The order of selection ofALF blocks is determined beforehand, and is in common with the selectionorder at the control information generating unit 112. Further, thecontrol unit 171 identifies the region of the ALF block to be processedwhich has been decided.

Since the image size of the frame is determined beforehand, upon the ALFblock size being determined, the number of ALF blocks necessary to paveALF blocks with the upper left of the frame as the point of origin(number of ALF blocks within the frame) can also be calculated. Thesetting values of the vertical direction size (number of pixels) andhorizontal direction size (number of pixels) of the ALF blocks areprovided beforehand, so the control unit 171 determines size of the ALFblocks and the number of ALF blocks following the setting values, andplaces the ALF blocks as to the decoded image.

Note that the number of ALF blocks is calculated by the followingExpression (4) and Expression (5).

num _(—) alf_block_(—) x=floor[(16×(pic_width_in_(—)mbs_minus1+1)+(alf_block_size−1))/alf_block_size]  (4)

num _(—) alf_block_(—)y=floor[(16×(pic_height_in_map_units_minus1+1)+(alf_block_size−1))/alf_block_size]  (5)

In Expression (4) and Expression (5), num_alf_block_x andnum_alf_block_y are the number of horizontal and vertical ALF blocksincluded in the image, respectively. Also, alf_block_size represents thesize of one side of an ALF block. To simplify description here, we willsay that ALF blocks are squares. Of course, an arrangement may be madewhere the vertical direction size and horizontal direction size of theALF blocks are different from each other.

The position of the i'th ALF block is expressed by the followingExpression (6) and Expression (7).

alf_block_(—) x=1%(num _(—) alf_block_(—) x−1)  (6)

alf_block_(—) y=floor[i/(num _(—) alf_block_(—) x−1)]  (7)

In Expression (6) and Expression (7), alf_block_x and alf_block_y eachindicate what number in the horizontal direction and vertical directionthe i'th ALF block is. The position of the upper left pixel of the i'thALF block is a position obtained by multiplying each of alf_block_x andalf_block_y by the alf_block_size. That is to say, the horizontaldirection is 16×alf_block_x, and the vertical direction is16×alf_block_y. Accordingly, the region of the i'th ALF block is a rangeof alf_block_size×alf_block_size from the upper left pixel thereof.

In step S175, the control unit 171 determines whether or not a region ofthe slice to be processed is included within the region of the ALF blockto be processed which ha been identified as described above. In theevent that determination is made that the region of the slice to beprocessed is included, the flow advances to step S176.

In step S176, the control unit 171 determines whether or not the valueof the filter block flag is 1. In the event that the value of the filterblock flag is 1, and instruction has been given to perform filterprocessing regarding the ALF block to be processed, control is effectedsuch that the selecting unit 173 selects the output of the filter 172,and the flow advances to step S177. In step S177, the control unit 171selects the pixels to be processed in a predetermined order such as, forexample, in raster scan order or the like, from unprocessed pixels.

In step S178, The control unit 171 determines whether or not pixels ofthe neighboring slice are necessary for filter processing of theselected pixel to be processed. In the event that pixels of theneighboring slice are included in the surrounding pixels of the pixel tobe processed, and determination is made that the pixel to be processedis a pixel near the slice boundary, the flow advances to step S179.

In step S179, the control unit 171 determines whether or not the valueof the boundary control value included in the control informationobtained by the control information generating unit 112 is “1”. In theevent that determination is made that the value of the boundary controlflag is “1”, the flow advances to step S180.

In step S180, the control unit 171 selects first adaptive filter forboundary 183 as the adaptive filter, and causes the first adaptivefilter for boundary 183 to perform filter processing straddling slicesas shown in A in FIG. 7. Upon the processing of step S180 ending, theflow advances to step S183.

Also, in step S179, in the event that determination is made that thevalue of the boundary control flag is “0”, the flow advances to stepS181.

In step S181, the control unit 171 selects the second adaptive filterfor boundary 184 as the adaptive filter, and causes the second adaptivefilter for boundary 184 to perform filter processing closed at thecurrent slice as shown in B in FIG. 7. Upon the processing of step S181ending, the flow advances to step S183.

Also, in the event that determination is made in step S178 that thepixel to be processed is not a pixel near a slice boundary, the flowadvances to step S182.

In step S182, the control unit 171 selects the in-slice adaptive filter182 as the adaptive filter, and causes the in-slice adaptive filter 182to perform normal filter processing using only pixels of the currentslice. Upon the processing in step S182 ending, the flow advances tostep S183.

Also, in the event that determination is made in step S176 that thevalue of the filter block flag is “02, the flow advances to step S183.Further, in the event that determination is made in step S175 that theALF block to be processed does not include the region of the slice to beprocessed, the flow advances to step S183.

In step S183, the control unit 171 determines whether all pixels withinthe ALF block to be processed have been processed. In the event thatdetermination is made that an unprocessed pixel exists, the flow returnsto step S177 and the subsequent processing is repeated.

Also, in the event that determination is made in step S183 that allpixels within the ALF block to be processed have been processed, theflow advances to step S184.

In step S184, the control unit 171 determines whether all ALF blockswithin the frame have been processed. In the event that determination ismade that an unprocessed ALF block exists, the flow returns to step S173and the subsequent processing is repeated. Also, in the event thatdetermination is made in step S184 that all ALF blocks have beenprocessed, adaptive filter control processing is ended, the flow returnsto step S111 in FIG. 9, and the processing of step S122 and on isperformed.

Note that the filter processing as to the pixels to be processed,performed in step S180 through step S182 are each executed independentlyas different tasks from the adaptive filter control processing. That isto say, in step S180 through step S182, upon executing of adaptivefilter processing being specified, the adaptive filter processing isexecuted as appropriate. That is to say, these adaptive filterprocessing are executable in parallel with the adaptive filter controlprocessing and adaptive filter processing as to other pixels.

These filter processing will be described with reference to theflowchart in FIG. 13 and FIG. 14. First, an example of the flow offilter processing executed by the first adaptive filter for boundary 183will be described with reference to the flowchart in FIG. 13.

Upon execution is instructed for filter processing straddling slices, ofwhich execution is instructed in step S180 in FIG. 12, in step S201 thefirst adaptive filter for boundary 183 monitors the buffer 181,determines whether or not all pixels of the surrounding region of thepixel to be processed are accumulated, and stands by until accumulated.Pixels of the surrounding region (i.e., surrounding pixels) includepixels of the neighboring slice as well. In the event that determinationis made that all the pixels have been accumulated in the buffer 181, theflow advances to step S202.

In step S202, the first adaptive filter for boundary 183 obtains pixelsof the surrounding region (surrounding pixels) from the buffer 181 andin step S203 uses the surrounding pixels and the filter coefficient setat the control unit 171 to perform filter processing of the pixel to beprocessed. Upon filter processing ending, the first adaptive filter forboundary 183 supplies the filter processing results to the selectingunit 173, and filter processing ends.

In the event that the value of the filter block flag is “1”, theselecting unit 173 selects the filter processing results, and suppliesto the frame memory 114 as the decoded image subjected to filteringprocessing, so as to be stored.

Next, an example of the flow of filter processing executed by the secondadaptive filter for boundary 184 will be described with reference to theflowchart in FIG. 14.

Upon execution is instructed to filter processing closed at the currentslice, of which execution is instructed in step S181 in FIG. 12, in stepS221 the second adaptive filter for boundary 184 duplicates surroundingpixels situated in the current slice already held in the buffer 181, andgenerates dummy data of the surrounding pixels situated in theneighboring slice.

Upon generating dummy data, in step S212 the second adaptive filter forboundary 184 uses the surrounding pixels including the dummy data andthe filter coefficient set at the control unit 171 to perform filterprocessing of the pixel to be processed. Upon the filter processingending, the second adaptive filter for boundary 184 supplies the filterprocessing results to the selecting unit 173 and the filter processingends.

In the event that the value of the filter block flag is “1”, theselecting unit 173 selects these filter processing results, and supplyto the frame memory 114 as the decoded image subjected to filterprocessing, so as to be stored.

As described above, based on the value of the boundary control flag, themethod for filter processing as a pixel near a boundary is selected asappropriate from multiple methods, whereby the adaptive filterprocessing unit 113 can suppress deterioration in the effects of filterprocessing due to local control of filter processing when encoding. Forexample, by performing filter processing so as to straddle slices, theadaptive filter processing unit 113 can improve the image quality of thefilter processing. Also, by performing filter processing closed at thecurrent slice, the adaptive filter processing unit 113 can performfilter processing with low delay.

At this time, the adaptive filter processing unit 113 selects the filterprocessing method based on the boundary control flag determined based onsystem specification information, so filter processing can be performedwithout breakdown of processing.

Also, the boundary control flag generating unit 132 sets a boundary flagbased on the system specification information, whereby the adaptivefilter processing unit 113 can be caused to execute filter processing soas to suppress deterioration of effects.

That is to say, the image encoding device 100 can suppress deteriorationin the effects of filter processing due to local control of filterprocessing when encoding.

Note that the lossless encoding unit 106 encodes the boundary controlflag and adds to the image compression information (embeds in the sliceheader, for example). Accordingly, the image encoding device 100 cancause an image decoding device which decodes the image compressioninformation output by the image encoding device 100 to suppressdeterioration in the effects of filter processing due to local controlof filter processing performed when decoding.

Now, to “add” means to correlate the boundary control flag to the imagecompression information with an optional form. For example, this may bedescribed as a syntax of the image compression information, or may bedescribe as user data. Also, the boundary control flag may be in a statelinked with the image compression information as metadata. That is tosay, to “add” includes “embedding”, “description”, “multiplexing”,“linking”, and so forth.

Also, with the above, description has been made to perform filterprocessing straddling slices or filter processing closed at the currentslice, as to pixels near the slice boundary, but filter processing maybe performed with other methods as well. Also, instead of performingfilter processing closed at the current slice, the filter processing maybe omitted, for example.

Further, it is sufficient for multiple filter processing methods for apixel near a slice boundary to have been prepared, and three or moremethods may be prepared as options. In this case, two bits or more arenecessary for the boundary control flag. Note that the number of bits ofthe boundary control flag is optional. However, the fewer the number ofbits are, the more the deterioration of encoding efficiency of the imagecompression information is suppressed, so unnecessarily increasing thenumber of bits is undesirable.

2. Second Embodiment [Configuration of Device]

Next, an image decoding device corresponding to the image encodingdevice 100 described with the first embodiment will be described. FIG.15 is a block diagram illustrating the configuration of an embodiment ofan image decoding device serving as an image processing device to whichthe present invention has been applied.

An image decoding device 200 decodes image compression informationoutput from the image encoding device 100, and generates a decodedimage.

An image decoding device 200 is configured of a storing buffer 201, alossless decoding unit 202, an inverse quantization unit 203, an inverseorthogonal transform unit 204, a computing unit 205, and a deblockingfilter 206. the image decoding device 200 also has an adaptive filterprocessing unit 207. The image decoding device 200 further has a screenrearranging buffer 208 and a D/A (Digital/Analog) conversion unit 209.The image decoding device 200 also has frame memory 210, an intraprediction unit 211, a motion compensation unit 212, and a selectingunit 213.

The storing buffer 201 stores a transmitted compressed imageinformation. The lossless decoding unit 202 decodes information suppliedfrom the storing buffer 201 and encoded by the lossless encoding unit106 in FIG. 1 using a format corresponding to the encoding format of thelossless encoding unit 106.

In the event that the current macroblock has been intra encoded, thelossless decoding unit 202 decodes the intra prediction mode informationstored in the header portion of the image compression information, andtransmits this information to the intra prediction unit 211. Also, inthe event that the current macroblock has been inter encoded, thelossless decoding unit 202 decodes the motion vector information storedin the header portion of the image compression information, andtransmits the information thereof to the motion compensation unit 212.

Also, the lossless decoding unit 202 extracts control information forthe adaptive filter (control information generated by the controlinformation generating unit 112) from the slice header of the imagecompression information, and decodes, and supplies the informationthereof to the adaptive filter processing unit 207.

The inverse quantization unit 203 subjects the image decoded by thelossless decoding unit 202 to inverse quantization using a formatcorresponding to the quantization format of the quantization unit 105 inFIG. 1. The inverse orthogonal transform unit 204 subjects the output ofthe inverse quantization unit 203 to inverse orthogonal transform usinga format corresponding to the orthogonal transform format of theorthogonal transform unit 104 in FIG. 1.

The computing unit 205 adds the prediction image supplied from theselecting unit 213 to the difference information subjected to inverseorthogonal transform, and generates a decoded image. The deblockingfilter 206 removes the block noise of the decoded image which has beengenerated by the adding processing.

The adaptive filter processing unit 207 performs filter processing onthe image supplied from the deblocking filter 206 based on the filtercoefficient, ALF block size, filter block flag, and boundary controlflag and the like, supplied from the lossless encoding unit. Theadaptive filter processing unit 207 performs adaptive filter processingin the same way as with the adaptive filter processing unit 113 inFIG. 1. Accordingly, the adaptive filter processing unit 207 can reduceblock noise and noise due to quantization which could not be completelyremoved with the deblocking filter 206.

The adaptive filter processing unit 207 supplies the image followingfilter processing to the frame memory 210 so as to be stored asreference image information, and also outputs to the screen rearrangingbuffer 208.

The screen rearranging buffer 208 performs rearranging of images. Thatis to say, the order of frames rearranged for encoding by the screenrearranging buffer 102 in FIG. 1 is rearranged to the original displayorder. The D/A conversion unit 209 performs D/A conversion of the imagesupplied from the screen rearranging buffer 208, and outputs. Forexample, the D/A conversion unit 209 outputs the output signals obtainedby performing D/A conversion to an unshown display, and displays animage.

The intra prediction unit 211 generates a prediction image based on theinformation supplied from the lossless decoding unit 202 in the eventthat the current frame has been intra encoded, and outputs the generatedprediction image to the selecting unit 213.

In the event that the current frame has been intra encoded, the motioncompensation unit 212 performs motion compensation processing as to thereference image information stored in the frame memory 210, based on themotion vector information supplied from the lossless decoding unit 202.

In the event that the current macroblock has been intra encoded, theselecting unit 213 connects to the intra prediction unit 211, andsupplies the image supplied from the intra prediction unit 211 to thecomputing unit 205 as a prediction image. Also, in the event that thecurrent macroblock has been inter encoded, the selecting unit 213connects to the motion compensation unit 212 and supplies the imagesupplied from the motion compensation unit 212 to the computing unit 205as a prediction image.

[Flow of Processing]

An example of the flow of decoding processing which this image decodingdevice 200 executes will be described with reference to the flowchart inFIG. 16.

In step S301, the storing buffer 201 stores the transmitted image. Instep S302, the lossless decoding unit 202 extracts the controlinformation for adaptive filter processing from the slice header of theimage compression information, and decodes this in step S303. Thedecoded control information is supplied to the adaptive filterprocessing unit 207.

Also, in step S303, the lossless decoding unit 202 decodes thecompressed image supplied from the storing buffer 201. Specifically, theI picture, P picture, and B picture encoded by the lossless encodingunit 106 in FIG. 1 are decoded.

At this time, the motion vector information, reference frameinformation, prediction mode information (information indicating theintra prediction mode or inter prediction mode), and so forth are alsodecoded.

Specifically, in the event that the prediction mode information is intraprediction mode information, the prediction mode information is suppliedto the intra prediction unit 211. In the event that the prediction modeinformation is inter prediction mode information, motion vectorinformation and reference frame information corresponding to theprediction mode information are supplied to the motion compensation unit212.

In step S304, the inverse quantization unit 203 inversely quantizes thetransform coefficient decoded in step S302 using a propertycorresponding to the property of the quantization unit 105 in FIG. 1. Instep S305, the inverse orthogonal transform unit 204 subjects thetransform coefficient inversely quantized in step S204 to inverseorthogonal transform using a property corresponding to the property ofthe orthogonal transform unit 104 in FIG. 1. This means that differenceinformation corresponding to the input of the orthogonal transform unit104 in FIG. 1 (the output of the computing unit 103) has been decoded.

In step S306, the computing unit 205 adds the prediction image selectedin the processing in later-described step S212 to the differenceinformation. Thus, the original image is decoded. In step S307, thedeblocking filter 206 subjects the image output from the computing unit205 to filtering. Thus, block noise is removed.

In step S308, the adaptive filter processing unit 207 performs adaptivefilter control processing for subjecting the image, subjected todeblocking filter processing, further to adaptive filter processing.This adaptive filter control processing is the same as the processingwhich the adaptive filter processing unit 113 in FIG. 1 performs. Thatis to say, this adaptive filter control processing is the same as thecase described with reference to the flowchart in FIG. 12, other thanusing the control information supplied from the lossless decoding unit202. Note however, the control information supplied from this losslessdecoding unit 202 has been generated by the control informationgenerating unit 112 in FIG. 1, and is substantially equivalent to thecontrol information supplied from the control information generatingunit 112 which the adaptive filter processing unit 113 in FIG. 1 uses.

Due to this adaptive filter control processing, block noise and noisedue to quantization which could not be completely removed with thedeblocking filter processing can be reduced.

In step S309, the frame memory 210 stores the image subjected tofiltering.

In the event that intra prediction mode information has been supplied,in step S310 the intra prediction unit 211 performs intra predictionprocessing in the intra prediction mode. Also, in the event that interprediction mode information has been supplied, in step S311 the motioncompensation unit 212 performs motion compensation processing in theintra prediction mode.

In step S312, the selecting unit 213 selects a prediction image. That isto say, one of the prediction image generated by the intra predictionunit 211 and the prediction image generated by the motion compensationunit 212 is selected, and the selected prediction image is supplied tothe computing unit 205.

For example, in the event of an image which has been intra encoded, theselecting unit 213 selects a prediction image generated by the intraprediction unit 211 and supplies this to the computing unit 205. Also,in the event of an image which as been inter encoded, the selecting unit213 selects a prediction image generated by the motion compensation unit212 and supplies this to the computing unit 205.

In step S313, the screen rearranging buffer 208 performs rearranging.Specifically, the sequence of frames rearranged for encoding by thescreen rearranging buffer 102 of the image encoding device 100 isrearranged to the original display sequence.

In step S314, the D/A conversion unit 209 performs D/A conversion of theimage from the screen rearranging buffer 208. This image is output to anunshown display, and the image is displayed.

Thus, with the image decoding unit 200, the lossless decoding unit 202extracts control information supplied from the image encoding device 100and decodes, and the adaptive filter processing unit 207 performsadaptive filter control processing (and filter processing) the same aswith the adaptive filter processing unit 113 of the image encodingdevice 100, using this control information.

By performing such adaptive filter control processing, the adaptivefilter processing unit 207 can suppress deterioration in the effects offilter processing due to local control of filter processing performedwhen decoding.

Accordingly, the image decoding device 200 can suppress deterioration inthe effects of filter processing due to local control of filterprocessing performed when decoding.

3. Third Embodiment [Image Processing System]

Note that while description has been made above that the systemspecification managing unit 141 of the control information generatingunit 112 holds or corrects system specification information, the systemspecification information may be made to include specificationinformation of the image decoding device.

In this case, in the event that the specification information of theimage decoding device is not known beforehand, the image encoding deviceneeds to collect the specification information of the image decodingdevice at a predetermined time, such as at the time of connectingcommunicably between the image encoding device and image decodingdevice, for example. At this time, the image encoding device may performcommunication with the image decoding to obtain the specificationinformation from the image decoding device, or specification input bythe user, for example, may be obtained.

Now, an unshown image processing system is a system where an imageencoding device 300 shown in FIG. 17 and an image decoding device 400shown in FIG. 18 are communicably connected via a communication mediumsuch as a network. The following is a description of the configurationof the devices.

FIG. 17 is a block diagram illustrating another example of an imageencoding device serving as an image processing device to which thepresent invention has been applied.

The image encoding device 300 is basically the same device as the imageencoding device 100 in FIG. 1, and has an image encoding unit 301.

The configuration of the image encoding unit 301 is the same as theconfiguration of the image encoding device 100, having the A/Dconversion unit 101 through rate control unit 119, and operates in thesame way as with the case described with the first embodiment.

Besides the image encoding unit 301, the image encoding device 300further has an input unit 302, communication unit 303, and informationcollecting unit.

The input unit 302 accepts operations of the user and the like. Thecommunication unit 303 performs communication with the image decodingdevice 400 via a network or the like. The information collecting unit304 collects specification information of the image decoding device 400input via the input unit 302 or specification information supplied fromthe image decoding device 400 via the communication unit 303. Theinformation collecting unit 304 supplies the collected specificationinformation to the system specification managing unit 141 of the controlinformation generating unit 112.

FIG. 18 is a block diagram illustrating another example of an imagedecoding device serving as an image processing device to which thepresent invention has been applied.

The image decoding device 400 is basically the same device as the imagedecoding device 200 in FIG. 15, and has an image decoding unit 401.

The configuration of the image decoding unit 401 is the same as theconfiguration of the image decoding device 200, having the storingbuffer 201 through selecting unit 213, and operates in the same way aswith the case described with the second embodiment.

Besides the image decoding unit 401, the image decoding device 400further has an information providing unit 402 and communication unit403.

The information providing unit 402 has specification information of theimage decoding device 400, and based on a request from the imageencoding device 300, provides the specification information. Thecommunication unit 403 performs communication with the image encodingdevice 300 via a network or the like. The communication unit 403 acceptsa request from the image encoding device 300, and supplies this to theinformation providing unit 402. the communication unit 403 also suppliesthe specification information of the image decoding device 400 suppliedfrom the information providing unit 402 in accordance with the requestto the image encoding device 300.

[Flow of Processing]

An example of the flow of exchange of specification information withsuch an image processing system will be described with reference to theflowchart in FIG. 19.

In step S401, the information collecting unit 304 of the image encodingdevice 300 requests the image decoding device 400 for specificationinformation of the image decoding device 400 via the communication unit303. Upon receiving the request in step S421, the communication unit 403of the image decoding device 400 supplies the request to the informationproviding unit 402.

In step S422, the information providing unit 402 supplies thespecification information of the image decoding device 400 to therequesting image encoding device 300 via the communication unit 403, asa response to the request.

Upon obtaining the specification information in step S402 via thecommunication unit 303, the information collecting unit 304 of the imageencoding device 300 supplies this to the system specification managingunit 141 of the control information generating unit 112.

In step S403, the image encoding unit 301 performs encoding processingbased on the specification information, and generates a code stream. Instep S404, the image encoding unit 301 supplies the generated codestream to the image decoding device 400.

In step S423, the image decoding unit 401 of the image decoding device400 obtains the code stream supplied from the image encoding device 300.In step S424, the image decoding unit 401 performs decoding processingas to the code stream.

Thus, specification information of the image decoding device 400 isexchanged before image encoding processing and image decodingprocessing, so the image encoding device 300 can create boundary controlflags based on the system specification information including thespecification information of the image decoding device 400.

Accordingly, the image encoding device 300 and the image decoding device400 can suppress deterioration in the effects of filter processing dueto local control of filter processing performed when encoding ordecoding, as described with the first embodiment and second embodiment.

4. Fourth Embodiment [Description of QALF]

ALF blocks may have a quad tree structure, as described with NPL 3. Thistechnique is called QALF (Quad tree-based Adaptive Loop Filter). A quadtree structure is a hierarchical structure where, at a lowerhierarchical level, the region of one ALF block one hierarchical levelabove is divided into four.

FIG. 20 illustrates an example where ALF block division is expressed bya quad tree structure where the maximum number of layers is three, witha filter block flag being specified for each ALF block.

A in FIG. 20 indicates a layer 0 which is an ALF block serving as theroot of the quad tree structure. In the quad tree structure, each ALFblock has a block partitioning flag indicating whether or not it isdivided into four at the lower hierarchical level. The value of theblock partitioning flag of the ALF block shown in A in FIG. 20 is “1”.That is to say, this ALF block is divided into four in the lowerhierarchical level (layer 1). B in FIG. 20 shows the layer 1. That is tosay, four ALF blocks are formed in the layer 1.

In the event that the block partitioning flag is “0”, a further lowerhierarchical level is not divided into four. That is to say, there is nofurther division, and a filter block flag is generated as to that ALFblock. That is to say, an ALF block of which the block partitioning flagis “0” also has a filter block flag. The “0” to the left of the “0-1”shown in B in FIG. 20 indicates the block partitioning flag of that ALFblock, and the “1” to the right shows the filter block flag of that ALFblock.

The two ALF blocks of which the block partitioning flag in layer 1 is“1” are divided into four in the lower hierarchical level (layer 2). Cin FIG. 20 illustrates the layer 2. That is to say, ten ALF blocks areformed in layer 2.

In the same way, ALF blocks with the block partitioning flag of “0” inlayer 2 are also assigned a filter block flag. In C in FIG. 20, theblock partitioning flag of one ALF block is “1”. That is to say, thatALF block is divided into four in the further lower hierarchical level(layer 3). D in FIG. 20 show the layer 3. That is to say, 13 ALF blocksare formed in the layer 3.

By forming a quad tree as shown in FIG. 20, the structure of the ALFblock ultimately becomes as shown in FIG. 21. Thus, with a quad treestructure, the size of ALF blocks differs with each hierarchical level.That is to say, by using a quad tree structure, the sizes of the ALFblocks can be made to be different one from another within the frame.

Control of the filter block flag in each ALF block is the same as withthe other embodiments described above. That is to say, filter processingis not performed in regions where the value of the filter block flag is“0” (the hatched portions in FIG. 21).

FIG. 22 illustrates an example of encoding the region of slice 1 in FIG.5 using the QALF technique. Here, the region of the heavy line 521represents the region of slice 1. Regardless of the ALF structure, theremay be cases where the surrounding pixels straddle multiple slices whenperforming filter processing on pixels near a slice boundary.Accordingly, the control method of filter processing as to pixels near aslice boundary can be performed in the same way as with theabove-described embodiments for the case of QALF as well.

That is to say, even with a case of quad tree structure ALF blocks, theimage encoding device and image decoding device can suppressdeterioration in the effects of filter processing due to local controlof filter processing performed when encoding or decoding.

5. Fifth Embodiment [Personal Computer]

The above-described series of processing may be executed by hardware,and may be executed by software. In this case, a configuration may bemade as a personal computer such as shown in FIG. 23, for example.

In FIG. 23, a CPU 601 of a personal computer 600 executes various typesof processing following programs stored in ROM (Read Only Memory) 602 orprograms loaded to RAM (Random Access Memory) 603 from a storage unit613. The RAM 603 also stores data and so forth necessary for the CPU 601to execute various types of processing, as appropriate.

The CPU 601, ROM 602, and RAM 603 are mutually connected by a bus 604.This bus 604 is also connected to an input/output interface 610.

Connected to the input/output interface 610 is an input unit 611 made upof a keyboard, a mouse, and so forth, an output unit 612 made up of adisplay such as a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display)or the like, a speaker, and so forth, a storage unit 613 made up of ahard disk and so forth, and a communication unit 614 made up of a modemand so forth. The communication unit 614 performs communicationprocessing via networks including the Internet.

Also connected to the input/output interface 610 is a drive 615 asnecessary, to which a removable medium 621 such as a magnetic disk, anoptical disc, a magneto-optical disk, semiconductor memory, or the like,is mounted as appropriate, and computer programs read out therefrom areinstalled in the storage unit 613 as necessary.

In the event of executing the above-described series of processing bysoftware, a program configuring the software is installed from a networkor recording medium.

This recording medium is not only configured of a removable medium 621made up of a magnetic disk (including flexible disk), optical disc(including CD-ROM (Compact Disc-Read Only Memory), DVD (DigitalVersatile Disc), magneto-optical disc (MD (Mini Disc)), or semiconductormemory or the like, in which programs are recorded and distributed so asto distribute programs to users separately from the device main unit,but also is configured of ROM 602, a hard disk included in the storageunit 613, and so forth, in which programs are recorded, distributed tousers in a state of having been built into the device main unitbeforehand.

Note that a program which the computer executes may be a program inwhich processing is performed in time sequence following the orderdescribed in the present Specification, or may be a program in whichprocessing is performed in parallel, or at a necessary timing, such aswhen a call-up has been performed.

Also, with the present Specification, steps describing programs recordedin the recording medium includes processing performed in time sequencefollowing the described order as a matter of course, and also processingexecuted in parallel or individually, without necessarily beingprocessed in time sequence.

Also, with the present specification, the term system represents theentirety of devices configured of multiple devices (devices).

Also, a configuration which has been described above as one device (orprocessing unit) may be divided and configured as multiple devices (orprocessing units). Conversely, configurations which have been describedabove as multiple devices (or processing units) may be integrated andconfigured as a single device (or processing unit). Also, configurationsother than those described above may be added to the devices (orprocessing units), as a matter of course. Further, part of aconfiguration of a certain device (or processing unit) may be includedin a configuration of another device (or another processing unit), aslong as the configuration and operations of the overall system issubstantially the same. That is to say, the embodiments of the presentinvention are not restricted to the above-described embodiments, andthat various modifications may be made without departing from theessence of the present invention.

For example, the above-described image encoding device 100, imagedecoding device 200, encoding device 300, and image decoding device 400may be applied to image various electronic devices. The following is adescription of examples thereof.

6. Sixth Embodiment [Television Receiver]

FIG. 24 is a block diagram illustrating a principal configurationexample of a television receiver using the image decoding device 200 orimage decoding device 400 to which the present invention has beenapplied.

A television receiver 1000 shown in FIG. 24 includes a terrestrial tuner1013, a video decoder 1015, a video signal processing circuit 1018, agraphics generating circuit 1019, a panel driving circuit 1020, and adisplay panel 1021.

The terrestrial tuner 1013 receives the broadcast wave signals of aterrestrial analog broadcast via an antenna, demodulates, obtains videosignals, and supplies these to the video decoder 1015. The video decoder1015 subjects the video signals supplied from the terrestrial tuner 1013to decoding processing, and supplies the obtained digital componentsignals to the video signal processing circuit 1018.

The video signal processing circuit 1018 subjects the video datasupplied from the video decoder 1015 to predetermined processing such asnoise removal or the like, and supplies the obtained video data to thegraphics generating circuit 1019.

The graphics generating circuit 1019 generates the video data of aprogram to be displayed on a display panel 1021, or image data due toprocessing based on an application to be supplied via a network, or thelike, and supplies the generated video data or image data to the paneldriving circuit 1020. Also, the graphics generating circuit 1019 alsoperforms processing such as supplying video data obtained by generatingvideo data (graphics) for the user displaying a screen used forselection of an item or the like, and superimposing this on the videodata of a program, to the panel driving circuit 1020 as appropriate.

The panel driving circuit 1020 drives the display panel 1021 based onthe data supplied from the graphics generating circuit 1019 to displaythe video of a program, or the above-mentioned various screens on thedisplay panel 1021.

The display panel 1021 is made up of an LCD (Liquid Crystal Display) andso forth, and displays the video of a program or the like in accordancewith the control by the panel driving circuit 1020.

Also, the television receiver 1000 also includes an audio A/D(Analog/Digital) conversion circuit 1014, an audio signal processingcircuit 1022, an echo cancellation/audio synthesizing circuit 1023, anaudio amplifier circuit 1024, and a speaker 1025.

The terrestrial tuner 1013 demodulates the received broadcast wavesignal, thereby obtaining not only a video signal but also an audiosignal. The terrestrial tuner 1013 supplies the obtained audio signal tothe audio A/D conversion circuit 1014.

The audio A/D conversion circuit 1014 subjects the audio signal suppliedfrom the terrestrial tuner 1013 to A/D conversion processing, andsupplies the obtained digital audio signal to the audio signalprocessing circuit 1022.

The audio signal processing circuit 1022 subjects the audio datasupplied from the audio A/D conversion circuit 1014 to predeterminedprocessing such as noise removal or the like, and supplies the obtainedaudio data to the echo cancellation/audio synthesizing circuit 1023.

The echo cancellation/audio synthesizing circuit 1023 supplies the audiodata supplied from the audio signal processing circuit 1022 to the audioamplifier circuit 1024.

The audio amplifier circuit 1024 subjects the audio data supplied fromthe echo cancellation/audio synthesizing circuit 1023 to D/A conversionprocessing, subjects to amplifier processing to adjust to predeterminedvolume, and then outputs the audio from the speaker 1025.

Further, the television receiver 1000 also includes a digital tuner1016, and an MPEG decoder 1017.

The digital tuner 1016 receives the broadcast wave signals of a digitalbroadcast (terrestrial digital broadcast, BS (Broadcasting Satellite)/CS(Communications Satellite) digital broadcast) via the antenna,demodulates to obtain MPEG-TS (Moving Picture Experts Group-TransportStream), and supplies this to the MPEG decoder 1017.

The MPEG decoder 1017 descrambles the scrambling given to the MPEG-TSsupplied from the digital tuner 1016, and extracts a stream includingthe data of a program serving as a playback object (viewing object). TheMPEG decoder 1017 decodes an audio packet making up the extractedstream, supplies the obtained audio data to the audio signal processingcircuit 1022, and also decodes a video packet making up the stream, andsupplies the obtained video data to the video signal processing circuit1018. Also, the MPEG decoder 1017 supplies EPG (Electronic ProgramGuide) data extracted from the MPEG-TS to a CPU 1032 via an unshownpath.

The television receiver 1000 uses the above-mentioned image decodingdevice 200 or image decoding device 400 as the MPEG decoder 1017 fordecoding video packets in this way. Note that the MPEG-TS transmittedfrom the broadcasting station or the like has been encoded by the imageencoding device 100 or image encoding device 300.

The MPEG decoder 1017 extracts and decodes control information suppliedfrom the image encoding device 100 or image encoding device 300, in thesame way as with the image decoding device 200 or image decoding device400, and performs adaptive filter control processing (and filterprocessing) using this control information. Accordingly, the MPEGdecoder 1017 can suppress deterioration in the effects of local controlof filter processing.

The video data supplied from the MPEG decoder 1017 is, in the same wayas with the case of the video data supplied from the video decoder 1015,subjected to predetermined processing at the video signal processingcircuit 1018, superimposed on the generated video data and so forth atthe graphics generating circuit 1019 as appropriate, supplied to thedisplay panel 1021 via the panel driving circuit 1020, and the imagethereof is displayed thereon.

The audio data supplied from the MPEG decoder 1017 is, in the same wayas with the case of the audio data supplied from the audio A/Dconversion circuit 1014, subjected to predetermined processing at theaudio signal processing circuit 1022, supplied to the audio amplifiercircuit 1024 via the echo cancellation/audio synthesizing circuit 1023,and subjected to D/A conversion processing and amplifier processing. Asa result thereof, the audio adjusted in predetermined volume is outputfrom the speaker 1025.

Also, the television receiver 1000 also includes a microphone 1026, andan A/D conversion circuit 1027.

The A/D conversion circuit 1027 receives the user's audio signalscollected by the microphone 1026 provided to the television receiver1000 serving as for audio conversation, subjects the received audiosignal to A/D conversion processing, and supplies the obtained digitalaudio data to the echo cancellation/audio synthesizing circuit 1023.

In the event that the user (user A)'s audio data of the televisionreceiver 1000 has been supplied from the A/D conversion circuit 1027,the echo cancellation/audio synthesizing circuit 1023 perform echocancellation with the user (user A)'s audio data taken as a object, andoutputs audio data obtained by synthesizing the user A's audio data andother audio data, or the like from the speaker 1025 via the audioamplifier circuit 1024.

Further, the television receiver 1000 also includes an audio codec 1028,an internal bus 1029, SDRAM (Synchronous Dynamic Random Access Memory)1030, flash memory 1031, a CPU 1032, a USB (Universal Serial Bus) I/F1033, and a network I/F 1034.

The A/D conversion circuit 1027 receives the user's audio signalcollected by the microphone 1026 provided to the television receiver1000 serving as for audio conversation, subjects the received audiosignal to A/D conversion processing, and supplies the obtained digitalaudio data to the audio codec 1028.

The audio codec 1028 converts the audio data supplied from the A/Dconversion circuit 1027 into the data of a predetermined format fortransmission via a network, and supplies to the network I/F 1034 via theinternal bus 1029.

The network I/F 1034 is connected to the network via a cable mounted ona network terminal 1035. The network I/F 1034 transmits the audio datasupplied from the audio codec 1028 to another device connected to thenetwork thereof, for example. Also, the network I/F 1034 receives, viathe network terminal 1035, the audio data transmitted from anotherdevice connected thereto via the network, and supplies this to the audiocodec 1028 via the internal bus 1029, for example.

The audio codec 1028 converts the audio data supplied from the networkI/F 1034 into the data of a predetermined format, and supplies this tothe echo cancellation/audio synthesizing circuit 1023.

The echo cancellation/audio synthesizing circuit 1023 performs echocancellation with the audio data supplied from the audio codec 1028taken as a object, and outputs the data of audio obtained bysynthesizing the audio data and other audio data, or the like, from thespeaker 1025 via the audio amplifier circuit 1024.

The SDRAM 1030 stores various types of data necessary for the CPU 1032performing processing.

The flash memory 1031 stores a program to be executed by the CPU 1032.The program stored in the flash memory 1031 is read out by the CPU 1032at predetermined timing such as when activating the television receiver1000, or the like. EPG data obtained via a digital broadcast, dataobtained from a predetermined server via the network, and so forth arealso stored in the flash memory 1031.

For example, MPEG-TS including the content data obtained from apredetermined server via the network by the control of the CPU 1032 isstored in the flash memory 1031. The flash memory 1031 supplies theMPEG-TS thereof to the MPEG decoder 1017 via the internal bus 1029 bythe control of the CPU 1032, for example.

The MPEG decoder 1017 processes the MPEG-TS thereof in the same way aswith the case of the MPEG-TS supplied from the digital tuner 1016. Inthis way, the television receiver 1000 receives the content data made upof video, audio, and so forth via the network, decodes using the MPEGdecoder 1017, whereby video thereof can be displayed, and audio thereofcan be output.

Also, the television receiver 1000 also includes a light reception unit1037 for receiving the infrared signal transmitted from a remotecontroller 1051.

The light reception unit 1037 receives infrared rays from the remotecontroller 1051, and outputs a control code representing the content ofthe user's operation obtained by demodulation, to the CPU 1032.

The CPU 1032 executes the program stored in the flash memory 1031 tocontrol the entire operation of the television receiver 1000 accordingto the control code supplied from the light reception unit 1037, and soforth. The CPU 1032, and the units of the television receiver 1000 areconnected via an unshown path.

The USB I/F 1033 performs transmission/reception of data as to anexternal device of the television receiver 1000 which is connected via aUSB cable mounted on a USB terminal 1036. The network I/F 1034 connectsto the network via a cable mounted on the network terminal 1035, alsoperforms transmission/reception of data other than audio data as tovarious devices connected to the network.

The television receiver 1000 uses the image decoding device 200 or imagedecoding device 400 as the MPEG decoder 1017, whereby deterioration inthe effects of local control of filter processing as to broadcastsignals received via an antenna or content data obtained via a networkcan be suppressed.

7. Seventh Embodiment [Cellular Telephone]

FIG. 25 is a block diagram illustrating a principal configurationexample of a cellular telephone using the image encoding device andimage decoding device to which the present invention has been applied.

A cellular telephone 1100 shown in FIG. 25 includes a main control unit1150 configured so as to integrally control the units, a power supplycircuit unit 1151, an operation input control unit 1152, an imageencoder 1153, a camera I/F unit 1154, an LCD control unit 1155, an imagedecoder 1156, a multiplexing/separating unit 1157, a recording/playbackunit 1162, a modulation/demodulation circuit unit 1158, and an audiocodec 1159. These are mutually connected via a bus 1160.

Also, the cellular telephone 1100 includes operation keys 1119, a CCD(Charge Coupled Devices) camera 1116, a liquid crystal display 1118, astorage unit 1123, a transmission/reception circuit unit 1163, anantenna 1114, a microphone (MIC) 1121, and a speaker 1117.

Upon a call end and power key being turned on by the user's operation,the power supply circuit unit 1151 activates the cellular telephone 1100in an operational state by supplying power to the units from a batterypack.

The cellular telephone 1100 performs various operations, such astransmission/reception of an audio signal, transmission/reception of ane-mail and image data, image shooting, data recoding, and so forth, invarious modes such as a voice call mode, a data communication mode, andso forth, based on the control of the main control unit 1150 made up ofa CPU, ROM, RAM, and so forth.

For example, in the voice call mode, the cellular telephone 1100converts the audio signal collected by the microphone (mike) 1121 intodigital audio data by the audio codec 1159, subjects this to spectrumspread processing at the modulation/demodulation circuit unit 1158, andsubjects this to digital/analog conversion processing and frequencyconversion processing at the transmission/reception circuit unit 1163.The cellular telephone 1100 transmits the signal for transmissionobtained by the conversion processing thereof to an unshown base stationvia the antenna 1114. The signal for transmission (audio signal)transmitted to the base station is supplied to the cellular telephone ofthe other party via the public telephone network.

Also, for example, in the voice call mode, the cellular telephone 1100amplifies the reception signal received at the antenna 1114, at thetransmission/reception circuit unit 1163, further subjects to frequencyconversion processing and analog/digital conversion processing, subjectsto spectrum inverse spread processing at the modulation/demodulationcircuit unit 1158, and converts into an analog audio signal by the audiocodec 1159. The cellular telephone 1100 outputs the converted andobtained analog audio signal thereof from the speaker 1117.

Further, for example, in the event of transmitting an e-mail in the datacommunication mode, the cellular telephone 1100 accepts the text data ofthe e-mail input by the operation of the operation keys 1119 at theoperation input control unit 1152. The cellular telephone 1100 processesthe text data thereof at the main control unit 1150, and displays on theliquid crystal display 1118 via the LCD control unit 1155 as an image.

Also, the cellular telephone 1100 generates e-mail data at the maincontrol unit 1150 based on the text data accepted by the operation inputcontrol unit 1152, the user's instructions, and so forth. The cellulartelephone 1100 subjects the e-mail data thereof to spectrum spreadprocessing at the modulation/demodulation circuit unit 1158, andsubjects to digital/analog conversion processing and frequencyconversion processing at the transmission/reception circuit unit 1163.The cellular telephone 1100 transmits the signal for transmissionobtained by the conversion processing thereof to an unshown base stationvia the antenna 1114. The signal for transmission (e-mail) transmittedto the base station is supplied to a predetermined destination via thenetwork, mail server, and so forth.

Also, for example, in the event of receiving an e-mail in the datacommunication mode, the cellular telephone 1100 receives the signaltransmitted from the base station via the antenna 1114 with thetransmission/reception circuit unit 1163, amplifies, and furthersubjects to frequency conversion processing and analog/digitalconversion processing. The cellular telephone 1100 subjects thereception signal thereof to spectrum inverse spread processing at themodulation/demodulation circuit unit 1158 to restore the original e-maildata. The cellular telephone 1100 displays the restored e-mail data onthe liquid crystal display 1118 via the LCD control unit 1155.

Note that the cellular telephone 1100 may record (store) the receivede-mail data in the storage unit 1123 via the recording/playback unit1162.

This storage unit 1123 is an optional rewritable recording medium. Thestorage unit 1123 may be semiconductor memory such as RAM, built-inflash memory, or the like, may be a hard disk, or may be a removablemedium such as a magnetic disk, a magneto-optical disk, an optical disc,USB memory, a memory card, or the like. It goes without saying that thestorage unit 1123 may be other than these.

Further, for example, in the event of transmitting image data in thedata communication mode, the cellular telephone 1100 generates imagedata by imaging at the CCD camera 1116. The CCD camera 1116 includes aCCD serving as an optical device such as a lens, diaphragm, and soforth, and serving as a photoelectric conversion device, which images asubject, converts the intensity of received light into an electricalsignal, and generates the image data of an image of the subject. The CCDcamera 1116 performs compression encoding of the image data at the imageencoder 1153 via the camera I/F unit 1154, and converts into encodedimage data.

The cellular telephone 1100 employs the above-mentioned image encodingdevice 100 or image encoding device 300 as the image encoder 1153 forperforming such processing. Accordingly, in the same way as with theimage encoding device 100 or image encoding device 300, the imageencoder 1053 can suppress deterioration of effects due to local controlof filter processing.

Note that, at this time simultaneously, the cellular telephone 1100converts the audio collected at the microphone (mike) 1121, whileshooting with the CCD camera 1116, from analog to digital at the audiocodec 1159, and further encodes this.

The cellular telephone 1100 multiplexes the encoded image data suppliedfrom the image encoder 1153, and the digital audio data supplied fromthe audio codec 1159 at the multiplexing/separating unit 1157 using apredetermined method. The cellular telephone 1100 subjects themultiplexed data obtained as a result thereof to spectrum spreadprocessing at the modulation/demodulation circuit unit 1158, andsubjects to digital/analog conversion processing and frequencyconversion processing at the transmission/reception circuit unit 1163.The cellular telephone 1100 transmits the signal for transmissionobtained by the conversion processing thereof to an unshown base stationvia the antenna 1114. The signal for transmission (image data)transmitted to the base station is supplied to the other party via thenetwork or the like.

Note that in the event that image data is not transmitted, the cellulartelephone 1100 may also display the image data generated at the CCDcamera 1116 on the liquid crystal display 1118 via the LCD control unit1155 instead of the image encoder 1153.

Also, for example, in the event of receiving the data of a moving imagefile linked to a simple website or the like in the data communicationmode, the cellular telephone 1100 receives the signal transmitted fromthe base station at the transmission/reception circuit unit 1163 via theantenna 1114, amplifies, and further subjects to frequency conversionprocessing and analog/digital conversion processing. The cellulartelephone 1100 subjects the received signal to spectrum inverse spreadprocessing at the modulation/demodulation circuit unit 1158 to restorethe original multiplexed data. The cellular telephone 1100 separates themultiplexed data thereof at the multiplexing/separating unit 1157 intoencoded image data and audio data.

The cellular telephone 1100 decodes the encoded image data at the imagedecoder 1156 using the decoding format corresponding to a predeterminedencoding format such as MPEG2, MPEG4, or the like, thereby generatingplayback moving image data, and displays this on the liquid crystaldisplay 1118 via the LCD control unit 1155. Thus, moving image dataincluded in a moving image file linked to a simple website is displayedon the liquid crystal display 1118, for example.

The cellular telephone 1100 employs the above-mentioned image decodingdevice 200 or image decoding device 400 as the image decoder 1156 forperforming such processing. Accordingly, in the same way as with theimage decoding device 200 or image decoding device 400, the imagedecoder 1156 extracts and decodes control information supplied from theimage encoding device 100 or image encoding device 300, and performsadaptive filter control processing (and filtering processing) using thecontrol information. Thus, the image decoder 1156 can suppressdeterioration of effects due to local control of filter processing.

At this time, simultaneously, the cellular telephone 1100 converts thedigital audio data into an analog audio signal at the audio codec 1159,and outputs this from the speaker 1117. Thus, audio data included in amoving image file linked to a simple website is played, for example.

Note that, in the same way as with the case of e-mail, the cellulartelephone 1100 may record (store) the received data linked to a simplewebsite or the like in the storage unit 1123 via the recording/playbackunit 1162.

Also, the cellular telephone 1100 analyzes the imaged two-dimensionalcode obtained by the CCD camera 1116 at the main control unit 1150,whereby information recorded in the two-dimensional code can beobtained.

Further, the cellular telephone 1100 can communicate with an externaldevice at the infrared communication unit 1181 using infrared rays.

The cellular telephone 1100 employs the image encoding device 100 orimage encoding device 300 as the image encoder 1153, whereby suppressioncan be realized of deterioration of effects due to local control offilter processing regarding encoded data generated by encoding imagedata generated at the CCD camera 1116, for example.

For example, the cellular telephone 1100 can improve the image qualityof filter processing results by performing filter processing straddlingslices, and can supply encoded data with higher image quality to othercellular telephones. Also, for example, by performing filter processingclosed at the current slice, the cellular telephone 1100 can performfilter processing with low delay, and can supply encoded data to othercellular telephones with lower delay.

Also, the cellular telephone 1100 employs the image decoding device 200or image decoding device 400 as the image decoder 1156, wherebysuppression can be realized of deterioration of effects due to localcontrol of filter processing regarding data of a moving image filelinked to at a simple website or the like, for example.

For example, the cellular telephone 1100 can improve the image qualityof filter processing results by performing filter processing straddlingslices, and can realize high image quality of decoded images. Also, forexample, by performing filter processing closed at the current slice,the cellular telephone 1100 can perform filter processing with lowdelay, and can decode encoded data with lower delay.

Note that description has been made so far wherein the cellulartelephone 1100 employs the CCD camera 1116, but the cellular telephone1100 may employ an image sensor (CMOS image sensor) using CMOS(Complementary Metal Oxide Semiconductor) instead of this CCD camera1116. In this case as well, the cellular telephone 1100 can image asubject and generate the image data of an image of the subject in thesame way as with the case of employing the CCD camera 1116.

Also, description has been made so far regarding the cellular telephone1100, but the image encoding device 100 and the image decoding device200 may be applied to any kind of device in the same way as with thecase of the cellular telephone 1100 as long as it is a device having thesame imaging function and communication function as those of thecellular telephone 1100, for example, such as a PDA (Personal DigitalAssistants), smart phone, UMPC (Ultra Mobile Personal Computer), netbook, notebook-sized personal computer, or the like.

8. Eighth Embodiment [Hard Disk Recorder]

FIG. 26 is a block diagram illustrating a principal configurationexample of a hard disk recorder which employs the image encoding deviceand image decoding device to which the present invention has beenapplied.

A hard disk recorder (HDD recorder) 1200 shown in FIG. 26 is a devicewhich stores, in a built-in hard disk, audio data and video data of abroadcast program included in broadcast wave signals (televisionsignals) received by a tuner and transmitted from a satellite or aterrestrial antenna or the like, and provides the stored data to theuser at timing according to the user's instructions.

The hard disk recorder 1200 can extract audio data and video data frombroadcast wave signals, decode these as appropriate, and store in thebuilt-in hard disk, for example. Also, the hard disk recorder 1200 canalso obtain audio data and video data from another device via thenetwork, decode these as appropriate, and store in the built-in harddisk, for example.

Further, the hard disk recorder 1200 can decode audio data and videodata recorded in the built-in hard disk, supply this to a monitor 1260,display an image thereof on the screen of the monitor 1260, and outputaudio thereof from the speaker of the monitor 1260, for example. Also,the hard disk recorder 1200 can decode audio data and video dataextracted from broadcast signals obtained via a tuner, or audio data andvideo data obtained from another device via a network, supply this tothe monitor 1260, display an image thereof on the screen of the monitor1260, and output audio thereof from the speaker of the monitor 1260, forexample.

Of course, operations other than these may be performed.

As shown in FIG. 26, the hard disk recorder 1200 includes a receptionunit 1221, a demodulation unit 1222, a demultiplexer 1223, an audiodecoder 1224, a video decoder 1225, and a recorder control unit 1226.The hard disk recorder 1200 further includes EPG data memory 1227,program memory 1228, work memory 1229, a display converter 1230, an OSD(On Screen Display) control unit 1231, a display control unit 1232, arecording/playback unit 1233, a D/A converter 1234, and a communicationunit 1235.

Also, the display converter 1230 includes a video encoder 1241. Therecording/playback unit 1233 includes an encoder 1251 and a decoder1252.

The reception unit 1221 receives the infrared signal from the remotecontroller (not shown), converts into an electrical signal, and outputsto the recorder control unit 1226. The recorder control unit 1226 isconfigured of, for example, a microprocessor and so forth, and executesvarious types of processing in accordance with the program stored in theprogram memory 1228. At this time, the recorder control unit 1226 usesthe work memory 1229 according to need.

The communication unit 1235, which is connected to the network, performscommunication processing with another device via the network. Forexample, the communication unit 1235 is controlled by the recordercontrol unit 1226 to communicate with a tuner (not shown), and toprincipally output a channel selection control signal to the tuner.

The demodulation unit 1222 demodulates the signal supplied from thetuner, and outputs to the demultiplexer 1223. The demultiplexer 1223separates the data supplied from the demodulation unit 1222 into audiodata, video data, and EPG data, and outputs to the audio decoder 1224,video decoder 1225, and recorder control unit 1226, respectively.

The audio decoder 1224 decodes the input audio data, and outputs to therecording/playback unit 1233. The video decoder 1225 decodes the inputvideo data, and outputs to the display converter 1230. The recordercontrol unit 1226 supplies the input EPG data to the EPG data memory1227 for storing.

The display converter 1230 encodes the video data supplied from thevideo decoder 1225 or recorder control unit 1226 into, for example, thevideo data conforming to the NTSC (National Television StandardsCommittee) format using the video encoder 1241, and outputs to therecording/playback unit 1233. Also, the display converter 1230 convertsthe size of the screen of the video data supplied from the video decoder1225 or recorder control unit 1226 into the size corresponding to thesize of the monitor 1260, converts the video data of which the screensize has been converted into the video data conforming to the NTSCformat using the video encoder 1241, converts into an analog signal, andoutputs to the display control unit 1232.

The display control unit 1232 superimposes, under the control of therecorder control unit 1226, the OSD signal output from the OSD (OnScreen Display) control unit 1231 on the video signal input from thedisplay converter 1230, and outputs to the display of the monitor 1260for display.

Also, the audio data output from the audio decoder 1224 has beenconverted into an analog signal using the D/A converter 1234, andsupplied to the monitor 1260. The monitor 1260 outputs this audio signalfrom a built-in speaker.

The recording/playback unit 1233 includes a hard disk as a recordingmedium in which video data, audio data, and so forth are recorded.

The recording/playback unit 1233 encodes the audio data supplied fromthe audio decoder 1224 by the encoder 1251. Also, the recording/playbackunit 1233 encodes the video data supplied from the video encoder 1241 ofthe display converter 1230 by the encoder 1251. The recording/playbackunit 1233 synthesizes the encoded data of the audio data thereof, andthe encoded data of the video data thereof using the multiplexer. Therecording/playback unit 1233 amplifies the synthesized data by channelcoding, and writes the data thereof in the hard disk via a recordinghead.

The recording/playback unit 1233 plays the data recorded in the harddisk via a playback head, amplifies, and separates into audio data andvideo data using the demultiplexer. The recording/playback unit 1233decodes the audio data and video data by the decoder 1252 using the MPEGformat. The recording/playback unit 1233 converts the decoded audio datafrom digital to analog, and outputs to the speaker of the monitor 1260.Also, the recording/playback unit 1233 converts the decoded video datafrom digital to analog, and outputs to the display of the monitor 1260.

The recorder control unit 1226 reads out the latest EPG data from theEPG data memory 1227 based on the user's instructions indicated by theinfrared signal from the remote controller which is received via thereception unit 1221, and supplies to the OSD control unit 1231. The OSDcontrol unit 1231 generates image data corresponding to the input EPGdata, and outputs to the display control unit 1232. The display controlunit 1232 outputs the video data input from the OSD control unit 1231 tothe display of the monitor 1260 for display. Thus, EPG (ElectronicProgram Guide) is displayed on the display of the monitor 1260.

Also, the hard disk recorder 1200 can obtain various types of data suchas video data, audio data, EPG data, and so forth supplied from anotherdevice via the network such as the Internet or the like.

The communication unit 1235 is controlled by the recorder control unit1226 to obtain encoded data such as video data, audio data, EPG data,and so forth transmitted from another device via the network, and tosupply this to the recorder control unit 1226. The recorder control unit1226 supplies the encoded data of the obtained video data and audio datato the recording/playback unit 1233, and stores in the hard disk, forexample. At this time, the recorder control unit 1226 andrecording/playback unit 1233 may perform processing such as re-encodingor the like according to need.

Also, the recorder control unit 1226 decodes the encoded data of theobtained video data and audio data, and supplies the obtained video datato the display converter 1230. The display converter 1230 processes, inthe same way as the video data supplied from the video decoder 1225, thevideo data supplied from the recorder control unit 1226, supplies to themonitor 1260 via the display control unit 1232 for displaying an imagethereof.

Alternatively, an arrangement may be made wherein in accordance withthis image display, the recorder control unit 1226 supplies the decodedaudio data to the monitor 1260 via the D/A converter 1234, and outputsaudio thereof from the speaker.

Further, the recorder control unit 1226 decodes the encoded data of theobtained EPG data, and supplies the decoded EPG data to the EPG datamemory 1227.

The hard disk recorder 1200 thus configured employs the image decodingdevice 200 or image decoding device 400 as the video decoder 1225,decoder 1252, and decoder housed in the recorder control unit 1226.Accordingly, in the same way as with the image decoding device 200 orimage decoding device 400, the video decoder 1225, decoder 1252, anddecoder housed in the recorder control unit 1226 extract and decodecontrol information supplied from the image encoding device 100 or imageencoding device 300, and perform adaptive filter control processing (andfilter processing) using the control information. Accordingly, the videodecoder 1225, decoder 1252, and decoder housed in the recorder controlunit 1226 can suppress deterioration of effects due to local control offilter processing.

Accordingly, the hard disk recorder 1200 can suppress deterioration ofeffects due to local control of filter processing regarding video datareceived via the tuner or communication unit 1235, and video datarecorded in the hard disk of the recording/playback unit 1233, forexample.

For example, the hard disk recorder 1200 can improve the image qualityof filter processing results by performing filter processing straddlingslices, and can realize high image quality of decoded images. Also, forexample, by performing filter processing closed at the current slice,the hard disk recorder 1200 can perform filter processing with lowdelay, and can decode encoded data with low delay.

Also, the hard disk recorder 1200 employs the image encoding device 100or image encoding device 300 as the encoder 1251. Accordingly, in thesame way as with the case of the image encoding device 100 or imageencoding device 300, the encoder 1251 can realize suppression ofdeterioration of effects due to local control of filter processing.

Accordingly, the hard disk recorder 1200 can suppress deterioration ofeffects due to local control of filter processing regarding encoded datarecorded in the hard disk, for example.

For example, the hard disk recorder 1200 can improve the image qualityof filter processing results by performing filter processing straddlingslices, and can record encoded data with higher image quality in thehard disk. Also, for example, by performing filter processing closed atthe current slice, the hard disk recorder 1200 can perform filterprocessing with low delay, and can generate encoded data and record inthe hard disk with lower delay.

Note that description has been made so far regarding the hard diskrecorder 1200 for recording video data and audio data in the hard disk,but it goes without saying that any kind of recording medium may beemployed. For example, even with a recorder to which a recording mediumother than a hard disk, such as flash memory, optical disc, video tape,or the like, is applied, the image encoding device 100 and imagedecoding device 200 can be applied thereto in the same way as with thecase of the above hard disk recorder 1200.

9. Ninth Embodiment [Camera]

FIG. 27 is a block diagram illustrating a principal configurationexample of a camera employing the image encoding device and imagedecoding device to which the present invention has been applied.

A camera 1300 shown in FIG. 27 images a subject, displays an image ofthe subject on an LCD 1316, and records this in a recording medium 1333as image data.

A lens block 1311 inputs light (i.e., picture of a subject) to aCCD/CMOS 1312. The CCD/CMOS 1312 is an image sensor employing a CCD orCMOS, which converts the intensity of received light into an electricalsignal, and supplies to a camera signal processing unit 1313.

The camera signal processing unit 1313 converts the electrical signalsupplied from the CCD/CMOS 1312 into color difference signals of Y, Cr,and Cb, and supplies to an image signal processing unit 1314. The imagesignal processing unit 1314 subjects, under the control of a controller1321, the image signal supplied from the camera signal processing unit1313 to predetermined image processing, or encodes the image signalthereof by an encoder 1341 using the MPEG format for example. The imagesignal processing unit 1314 supplies encoded data generated by encodingan image signal, to a decoder 1315. Further, the image signal processingunit 1314 obtains data for display generated at an on-screen display(OSD) 1320, and supplies this to the decoder 1315.

With the above-mentioned processing, the camera signal processing unit1313 appropriately takes advantage of DRAM (Dynamic Random AccessMemory) 1318 connected via a bus 1317 to hold image data, encoded dataencoded from the image data thereof, and so forth in the DRAM 1318thereof according to need.

The decoder 1315 decodes the encoded data supplied from the image signalprocessing unit 1314, and supplies obtained image data (decoded imagedata) to the LCD 1316. Also, the decoder 1315 supplies the data fordisplay supplied from the image signal processing unit 1314 to the LCD1316. The LCD 1316 synthesizes the image of the decoded image data, andthe image of the data for display, supplied from the decoder 1315 asappropriate, and displays a synthesizing image thereof.

The on-screen display 1320 outputs, under the control of the controller1321, data for display such as a menu screen or icon or the like made upof a symbol, characters, or a figure to the image signal processing unit1314 via the bus 1317.

Based on a signal indicating the content commanded by the user using anoperating unit 1322, the controller 1321 executes various types ofprocessing, and also controls the image signal processing unit 1314,DRAM 1318, external interface 1319, on-screen display 1320, media drive1323, and so forth via the bus 1317. A program, data, and so forthnecessary for the controller 1321 executing various types of processingare stored in FLASH ROM 1324.

For example, the controller 1321 can encode image data stored in theDRAM 1318, or decode encoded data stored in the DRAM 1318 instead of theimage signal processing unit 1314 and decoder 1315. At this time, thecontroller 1321 may perform encoding and decoding processing using thesame format as the encoding and decoding format of the image signalprocessing unit 1314 and decoder 1315, or may perform encoding anddecoding processing using a format that neither the image signalprocessing unit 1314 nor the decoder 1315 can handle.

Also, for example, in the event that start of image printing has beeninstructed from the operating unit 1322, the controller 1321 reads outimage data from the DRAM 1318, and supplies this to a printer 1334connected to the external interface 1319 via the bus 1317 for printing.

Further, for example, in the event that image recording has beeninstructed from the operating unit 1322, the controller 1321 reads outencoded data from the DRAM 1318, and supplies this to a recording medium1333 mounted on the media drive 1323 via the bus 1317 for storing.

The recording medium 1333 is an optional readable/writable removablemedium, for example, such as a magnetic disk, a magneto-optical disk, anoptical disc, semiconductor memory, or the like. It goes without sayingthat the recording medium 1333 is also optional regarding the type of aremovable medium, and accordingly may be a tape device, or may be adisc, or may be a memory card. It goes without saying that the recodingmedium 1333 may be a non-contact IC card or the like.

Alternatively, the media drive 1323 and the recording medium 1333 may beconfigured so as to be integrated into a non-transportability recordingmedium, for example, such as a built-in hard disk drive, SSD (SolidState Drive), or the like.

The external interface 1319 is configured of, for example, a USBinput/output terminal and so forth, and is connected to the printer 1334in the event of performing printing of an image. Also, a drive 1331 isconnected to the external interface 1319 according to need, on which theremovable medium 1332 such as a magnetic disk, optical disc, ormagneto-optical disk is mounted as appropriate, and a computer programread out therefrom is installed in the FLASH ROM 1324 according to need.

Further, the external interface 1319 includes a network interface to beconnected to a predetermined network such as a LAN, the Internet, or thelike. For example, in accordance with the instructions from theoperating unit 1322, the controller 1321 can read out encoded data fromthe DRAM 1318, and supply this from the external interface 1319 toanother device connected via the network. Also, the controller 1321 canobtain, via the external interface 1319, encoded data or image datasupplied from another device via the network, and hold this in the DRAM1318, or supply this to the image signal processing unit 1314.

The camera 1300 thus configured employs the image decoding device 200 orimage decoding device 400 as the decoder 1315. Accordingly, in the sameway as with the image decoding device 200 or image decoding device 400,the decoder 1315 extracts and decodes control information supplied fromthe image encoding device 100 or image encoding device 300, and performsadaptive filter control processing (and filter processing) using thecontrol information. Accordingly, the decoder 1315 can suppressdeterioration of effects due to local control of filter processing.

Accordingly, the camera 1300 can suppress deterioration of effects dueto local control of filter processing regarding, for example, from theimage data generated at the CCD/CMOS 1312, the encoded data of videodata read out from the DRAM 1318 or recording medium 1333, and encodeddata of video data obtained via the network.

For example, the camera 1300 can improve the image quality of filterprocessing results by performing filter processing straddling slices,and can realize high image quality of decoded images. Also, for example,by performing filter processing closed at the current slice, the camera1300 can perform filter processing with low delay, and can decodeencoded data with low delay.

Also, the camera 1300 employs the image encoding device 100 or imageencoding device 300 as the encoder 1341. Accordingly, in the same way aswith the case of the image encoding device 100 or image encoding device300, the encoder 1341 can realize suppression of deterioration ofeffects due to local control of filter processing.

Accordingly, the camera 1300 can suppress deterioration of effects dueto local control of filter processing regarding the encoded datarecorded in the DRAM 1318 or recording medium 1333, or encoded data tobe provided to other devices, for example.

For example, the camera 1300 can improve the image quality of filterprocessing results by performing filter processing straddling slices,and can record encoded data with higher image quality in the DRAM 1318or recording medium 1333, or provide this to other devices. Also, forexample, by performing filter processing closed at the current slice,the camera 1300 can perform filter processing with low delay, and cangenerate encoded data and record in the in the DRAM 1318 or recordingmedium 1333, or provide this to other devices, with lower delay.

Note that the decoding method of the image decoding device 200 or imagedecoding device 400 may be applied to the decoding processing which thecontroller 1321 performs. In the same way, the encoding method of theimage encoding device 100 or image encoding device 300 may be applied tothe encoding processing which the controller 1321 performs.

Also, the image data which the camera 1300 takes may be moving images ormay be still images.

As a matter of course, the image encoding device 100, image decodingdevice 200, image encoding device 300, and image decoding device 400 maybe applied to devices or systems other than the above-described devices.

Also, the size of macroblocks is not restricted to 16×16 pixels.Application can be made to macroblocks of various sizes, such as that of32×32 pixels shown in FIG. 28, for example.

While description has been made above with flag information and the likebeing multiplexed (described) in the bit stream, flags and image data(or bit stream) may be transmitted (recorded), for example, besidesbeing multiplexed. A form may be made where the flag and image data (orbit stream) are linked (added) as well.

Linking (adding) indicates a state in which image data (or bit streams)and flags are mutually linked (a correlated state), and the physicalpositional relation is arbitrary. For example, the image data (or bitstream) and flags may be transmitted over separate transmission paths.Also, the image data (or bit stream) and flags may each be recorded inseparate recording mediums (or in separate recording areas within thesame recording medium). Note that the increments in which image data (orbit streams) and flags are linked are optional, and may be set inincrements of encoding processing (one frame, multiple frames, etc.),for example.

REFERENCE SIGNS LIST

-   -   100 image encoding device    -   112 control information generating unit    -   113 adaptive filter control unit    -   132 boundary control flag generating unit    -   141 system specification managing unit    -   142 determining unit    -   161 pixel to be processed    -   162 surrounding pixels    -   163 slice boundary    -   171 control unit    -   172 adaptive filter    -   173 selecting unit    -   181 buffer    -   182 in-slice adaptive filter    -   183 first adaptive filter for boundary    -   184 second adaptive filter for boundary    -   200 image decoding device    -   202 lossless decoding unit    -   207 adaptive filter processing unit    -   300 image encoding device    -   301 image encoding unit    -   302 input unit    -   303 communication unit    -   304 information collection unit    -   400 image decoding unit    -   401 image decoding unit    -   402 information providing unit    -   403 communication unit

1. An image processing device comprising: determining means configuredto determine whether or not there are included, in surrounding pixels ofa pixel to be processed by filter processing locally performed on animage, pixels of a slice neighboring a slice in which said pixel to beprocessed is included; selecting means configured to select, from aplurality of methods, a method for said filter processing to beperformed on said pixel to be processed, based on a boundary controlflag, in the event that determination has been made by said determiningmeans that a pixel of said neighboring slice is included in saidsurrounding pixels; and filter processing means configured to performsaid filter processing as to said pixel to be processed with the methodselected by said selecting means.
 2. The image processing deviceaccording to claim 1, wherein said selecting means select one of amethod to perform said filter processing on said pixel to be processedafter said surrounding pixels situated in said neighboring slice havebeen obtained, and a method to perform said filter processing on saidpixel to be processed by generating dummy data of said surroundingpixels situated in said neighboring slice by duplicating saidsurrounding pixels situated in the slice including said pixel to beprocessed.
 3. The image processing device according to claim 1, whereinsaid selecting means select one of a method to perform said filterprocessing on said pixel to be processed after said surrounding pixelssituated in said neighboring slice have been obtained, and a method toomit performing said filter processing on said pixel to be processed. 4.The image processing device according to claim 1, further comprising:generating means configured to generate said boundary control flag basedon system specifications; wherein said selecting means select a methodof said filter processing as to said pixel to be processed, based onsaid boundary control flag generated by said generating means.
 5. Theimage processing device according to claim 4, wherein said systemspecifications include hardware resources of said image processingdevice.
 6. The image processing device according to claim 4, whereinsaid system specifications include the usage purpose of said imageprocessing device.
 7. The image processing device according to claim 4,further comprising: encoding means configured to encode said image andgenerate encoded data; wherein said encoding means further encode saidboundary control flag generated by said generating means, and add tosaid encoded data.
 8. The image processing device according to claim 1,further comprising: decoding means configured to decode encoded data ofsaid image having been encoded, and generate said image; wherein saiddecoding means further decode said encoded boundary control flag whichhas been added to said encoded data; and wherein said selecting meansselect a method for said filter processing as to said pixel to beprocessed, based on said boundary control flag decoded by said decodingmeans.
 9. An image processing method wherein determining means of animage processing device determine whether or not there are included, insurrounding pixels of a pixel to be processed by filter processinglocally performed on an image, pixels of a slice neighboring a slice inwhich said pixel to be processed is included, selecting means of saidimage processing device select, from a plurality of methods, a methodfor said filter processing to be performed on said pixel to beprocessed, based on a boundary control flag, in the event thatdetermination has been made that a pixel of said neighboring slice isincluded in said surrounding pixels, and filter processing means of saidimage processing device perform said filter processing as to said pixelto be processed with the method that has been selected.